Rescue mode: fixing a server you locked yourself out of

Every administrator locks themselves out eventually: a typo in sshd_config, a firewall rule that included port 22 in the deny list, a disk that filled up overnight, a line in fstab pointing at a volume that no longer exists. Rescue mode exists for exactly those mornings. It boots a live system from the network with your disk untouched, so you can mount it, fix the one line that broke, and reboot into a working server. This is how to use it, and the four fixes that cover most cases.

What rescue mode does

From the server's Power tab, Boot into rescue reboots the server into a small Debian-based image loaded over the network. Your disks are not modified or mounted; the rescue system runs from RAM. It has SSH with your panel keys pre-installed, the usual tools (mount, chroot, fsck, mdadm, cryptsetup, zfs, editors), and network access. The console in the panel works too if SSH is what you broke. Leaving rescue is a normal reboot; the panel shows a banner while rescue is active so you do not forget.

Before anything else, take a snapshot from the panel if you can still get one. If you cannot (the server is already in rescue), the disk is still safe until you write to it; work carefully.

Mounting your disk

lsblk -f                                # find the root partition: usually vda2 (VPS) or md0 / nvme0n1p2 (dedicated)
mount /dev/vda2 /mnt
mount /dev/vda1 /mnt/boot               # if /boot is separate; on UEFI images: /mnt/boot/efi
for d in dev proc sys run; do mount --rbind /$d /mnt/$d; done
chroot /mnt /bin/bash                   # you are now "inside" your server, without running it

On a dedicated server with a software mirror, run mdadm --assemble --scan first and mount /dev/md0; on ZFS, zpool import -f -R /mnt rpool mounts everything in one go.

Fix 1: a broken sshd config

sshd -t -f /mnt/etc/ssh/sshd_config      # from outside the chroot: prints the offending line
nano /mnt/etc/ssh/sshd_config.d/10-hardening.conf

The usual culprits: a Match block that swallowed the options below it, PasswordAuthentication no before the key was installed, or a port change without the firewall rule. Fix the line, test again with sshd -t, reboot.

Fix 2: a firewall rule that bit you

chroot /mnt ufw disable                 # ufw: turns it off at next boot; re-enable after fixing the rule
nano /mnt/etc/nftables.conf             # nftables: remove or fix the rule, keep the file valid
rm /mnt/etc/iptables/rules.v4           # iptables-persistent: drop the saved rules entirely

Reboot, log in, add the missing allow 22/tcp (or your port), re-enable. If you use a cloud-init or Ansible-managed firewall, fix the source too, or the next run will lock you out again.

Fix 3: a full disk

du -xh --max-depth=2 /mnt | sort -h | tail -20    # where did it go?
journalctl --root=/mnt --vacuum-size=200M          # systemd journal, a common offender
rm -rf /mnt/var/lib/docker/tmp/*                   # or prune from inside after boot
find /mnt/var/log -name '*.gz' -delete             # rotated logs are safe to remove

Free a few gigabytes, reboot, and then fix the cause: log rotation, a backup script writing locally, a database without a retention policy. A disk that hits 100% will do it again next month if the cause stays.

Fix 4: a bad fstab line

A missing volume in /etc/fstab without the nofail option stops the boot at an emergency shell you cannot reach. Comment the line out, or add nofail,x-systemd.device-timeout=10 to it, and reboot. Every non-root mount should carry nofail; it is the one fstab habit that prevents this class of lockout entirely.

Other things rescue is good for

• Resetting a root password: chroot /mnt passwd. The panel can do it without rescue, but not if you changed the SSH key layout underneath it.
• Running fsck on a file system that refuses to mount: unmounted, from rescue, is the only safe place to do it.
• Copying data off a server you are abandoning: rsync -a /mnt/srv/ other-server:/srv/ over the rescue system's network.
• Regenerating the bootloader after a kernel package went wrong: inside the chroot, update-grub or grub-install (BIOS) / update-initramfs -u.
• Investigating a compromise with the operating system not running, so nothing on it can hide from you.

Leaving rescue

exit                                     # leave the chroot
umount -R /mnt                          # or: zpool export rpool
reboot

The panel's Power tab has an Exit rescue button that does the same if the shell is gone. The server boots from its own disk again, and the event log records the rescue session with its duration.

Before next time

Snapshot before risky changes; it turns a rescue into a rollback. Test a firewall change with ufw's rule inserted while a second session stays open. Never close the working SSH session until the new one works. And keep the panel's two-factor set up: rescue mode is a powerful door, and the panel is the lock on it.