grub rescue

Either the first or second major hurdle you had to clear back when I took the RHCE test was recovering a system whose /boot partition had been blown away. Deja vu.

The home file server apparently rebooted today as the result of a series of brownouts in the neighborhood. Hey, thanks Duke Energy!

Since the machine’s only real job is to host backup files no one noticed until I got home and tried logging in to check for updates.

No route to host.

After attaching a monitor to the normally headless beast I was greeted with a “grub rescue >” prompt, which some sources on the Internet assure us is much better than the old “grub >”.

Speak for yourselves.

I found that most of the wisdom out on the cloud wasn’t aligned with the CentOS 7 system I’m running — in other words, is out of date and useless (except as a means of confusing people).

In the end I decided to use the good old-fashioned rescue disk method (boot with a CentOS 7 install usb image, select Troubleshooting and then do a “chroot /mnt/sysimage” when I got a shell prompt).

What I found was that /boot had either been deleted or corrupted. As luck would have it I had taken a backup of /boot yesterday as part of testing duply/duplicity, and so was able to restore it to my /data/restore folder and then copy its contents into the seemingly empty /boot. I then ran “grub2-install /dev/sda” to re-install grub’s boot code on the system.

Upon rebooting my system came up without any apparent ill effects (except for a debugging menu item for each of the kernels installed).

In looking for answers during this “crisis”, I discovered the subject wasn’t covered in any of the RHCE study guides and that there were no RHEL 7 specific doc that provided a procedure to follow. Fortunately, I remembered enough from the RHCE class to get by, adjusting for the shift from mere grub to “grub2” along the way.

So I guess the big question now is, what disk imaging solution should I used to provide a way to quickly perform a bare metal recovery of my Linux systems?

This entry was posted in System Administration, Systems Analysis on by .

About phil

My name is Phil Lembo. In my day job I’m an enterprise IT architect for a leading distribution and services company. The rest of my time I try to maintain a semi-normal family life in the suburbs of Raleigh, NC. E-mail me at philipATlembobrothersDOTcom. The opinions expressed here are entirely my own and not those of my employers, past, present or future (except where I quote others, who will need to accept responsibility for their own rants).