Today we had a server stalled on “Loading kernel modules” at reboot (after adding 12GB of RAM, to 24GB total). The datacenter didn’t know what to do and they put us on a 32-bit rescue mode console from which we couldn’t (obviously) launch a 64bit chroot to update the kernel. The situation seemed pretty desperate.
Our sysadmin, Jérôme, once again came to the rescue. Waiting for the datacenter to respond would have potentially increased reboot time up to 45 minutes. The only possible thing to do: replace initrd with a similar version (just in case the first one would have been damaged). Luckily, we had another similar machine which had been installed the sameday (at 1h difference), so we copied the initrd image over and launched a reboot. After 5 long (never-ending) minutes of no response from the server, it just popped back online.
That’s another one to remember for sysadmin’s day!