At work, we’ve recently had problems with one of our SANS, and as a result we ended up with some filesystem corruption and a little data loss.
As part of our clean-up effort, we rebooted and checked each server, mainly by
running the classic
shutdown -F -r now, to force a reboot and
systems where there’s little or no damage, this does exactly what you’d expect,
and you end up with the system coming back up happy, but on some CentOS 7
systems where there was corruption this is where the fun began.
Like on all modern Linux systems, if a filesystem check fails you’ll be dropped
into a Maintenance Shell (or rather, prompted for the root password, and then
given a shell). This shell is designed to let you do some diagnostics and in
the case of filesystem problems the main thing you’re going to want to do is
On CentOS 7 systems, you’re dropped into something called rescue mode, which is a systemd unit that does almost everything you want, except:
In rescue mode, the system attempts to mount all local file systems and start some important system services, but it does not activate network interfaces or allow more users to be logged into the system at the same time.
That’s right, it mounts the filesystems! This makes it hard to check them,
especially if you have errors on
auditd uses. In fact, even
fuser —k m /var failed to free the filesystem for unmounting.
I think this is particularly annoying, considering that the one of the main uses
of Maintenance Modes is filesystem repair, so I set about searching the web for
the answer, but despite searching for all manner of things like
recovery mode fsck and
rhel 7 fsck recovery, I didn’t really get anywhere,
Luckily, I found a page on
which controls the behaviour of
fsck on startup, and has the snippet:
fsck.repair= One of “preen”, “yes”, “no”. Controls the mode of operation. The default is “ preen”, and will automatically repair problems that can be safely fixed. “yes ” will answer yes to all questions by fsck and “no” will answer no to all questions.
Perfect! So, you can force the system to
fsck and repair by appending
fsck.repair=yes to the kernel command line in grub. Handy! Hopefully my
posting this will help some poor soul with the same problems as I had.