Either the first or second major hurdle you had to clear back when I took the RHCE test was recovering a system whose /boot partition had been blown away. Deja vu.
The home file server apparently rebooted today as the result of a series of brownouts in the neighborhood. Hey, thanks Duke Energy!
Since the machine’s only real job is to host backup files no one noticed until I got home and tried logging in to check for updates.
No route to host.
After attaching a monitor to the normally headless beast I was greeted with a “grub rescue >” prompt, which some sources on the Internet assure us is much better than the old “grub >”.
Speak for yourselves.
I found that most of the wisdom out on the cloud wasn’t aligned with the CentOS 7 system I’m running — in other words, is out of date and useless (except as a means of confusing people).
In the end I decided to use the good old-fashioned rescue disk method (boot with a CentOS 7 install usb image, select Troubleshooting and then do a “chroot /mnt/sysimage” when I got a shell prompt).
What I found was that /boot had either been deleted or corrupted. As luck would have it I had taken a backup of /boot yesterday as part of testing duply/duplicity, and so was able to restore it to my /data/restore folder and then copy its contents into the seemingly empty /boot. I then ran “grub2-install /dev/sda” to re-install grub’s boot code on the system.
Upon rebooting my system came up without any apparent ill effects (except for a debugging menu item for each of the kernels installed).
In looking for answers during this “crisis”, I discovered the subject wasn’t covered in any of the RHCE study guides and that there were no RHEL 7 specific doc that provided a procedure to follow. Fortunately, I remembered enough from the RHCE class to get by, adjusting for the shift from mere grub to “grub2” along the way.
So I guess the big question now is, what disk imaging solution should I used to provide a way to quickly perform a bare metal recovery of my Linux systems?