Linux System Rescue & Recovery — Boot Repair, Data Recovery & Emergency Procedures

In this tutorial, you'll learn about Linux System Rescue & Recovery. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Linux system rescue covers the procedures and tools needed to recover a non-booting system, repair broken bootloaders, fix filesystem corruption, reset lost passwords, and recover data from failing drives.

What You'll Learn

How to boot into recovery mode, chroot into a broken system, repair GRUB, fix filesystem issues with fsck, recover deleted files, use ddrescue on failing drives, and build a custom rescue USB with essential tools.

Why Rescue Skills Matter

Every administrator faces a system that will not boot — corrupted GRUB, failed kernel update, full root partition, or disk errors. Without rescue skills, the only option is reinstalling. With tools like chroot, fsck, and ddrescue, most systems can be repaired in minutes. Doda Browser's QA infrastructure uses automated rescue scripts that chroot into broken CI nodes and restore them from snapshots.

Learning Path

flowchart LR
  A[Performance Tuning] --> B[System Rescue
You are here]
  B --> C[journalctl Guide]
  B --> D[Backup Strategies]
  style B fill:#f90,color:#fff

The Rescue Toolbox

Every rescue scenario starts with a live Linux environment. Create a multi-purpose rescue USB:

# Build a custom rescue USB
sudo apt install live-build
sudo lb config --distribution jammy --archive-areas "main universe multiverse"
echo "live-task-recovery live-task-forensics" >> config/package-lists/rescue.list.chroot
sudo lb build

# Or use a general-purpose live USB with tools pre-loaded
# Tools to have: fsck, ddrescue, testdisk, photorec, rsync, lvm2, mdadm, ntfs-3g, xfsprogs, btrfs-progs

Recovery Mode Boot

Ubuntu's recovery menu and GRUB's advanced options provide built-in rescue access:

# At GRUB menu:
# 1. Press 'e' to edit boot parameters
# 2. Find the line starting with "linux" and append:
#    systemd.unit=rescue.target
#    or: single
#    or: init=/bin/bash
# 3. Press Ctrl+X to boot

# Alternative: append rw to mount root read-write
# linux /vmlinuz-... root=/dev/sda1 rw single

Chroot Rescue Procedure

Chroot lets you use a live environment to fix your installed system:

# Boot from a live USB, then:

# 1. Identify and mount the root partition
lsblk
sudo mount /dev/sda1 /mnt

# 2. Mount necessary virtual filesystems
sudo mount --bind /dev /mnt/dev
sudo mount --bind /proc /mnt/proc
sudo mount --bind /sys /mnt/sys
sudo mount --bind /run /mnt/run

# For UEFI systems, mount EFI partition
sudo mount /dev/sda2 /mnt/boot/efi

# 3. Chroot into the system
sudo chroot /mnt /bin/bash
source /etc/profile
export PS1="(chroot) \w # "

# 4. Now you can run repair commands inside the broken system

What to Do Inside Chroot

# Reinstall GRUB
grub-install /dev/sda
update-grub

# Fix broken packages
apt update && apt install --fix-broken
apt dist-upgrade

# Reinstall a broken kernel
apt install --reinstall linux-image-$(uname -r)

# Reset root password
passwd root

# Rebuild initramfs
update-initramfs -u -k all

# Fix filesystem fstab issues
blkid
nano /etc/fstab

# Rebuild LVM if volume groups are missing
vgscan --mknodes
vgchange -ay

# Exit and reboot
exit
sudo umount -R /mnt
sudo reboot

GRUB Recovery

When GRUB itself is broken:

# From GRUB rescue prompt:
grub rescue> set prefix=(hd0,msdos1)/boot/grub
grub rescue> set root=(hd0,msdos1)
grub rescue> insmod normal
grub rescue> normal

# Once booted, reinstall GRUB from the OS:
sudo grub-install /dev/sda
sudo update-grub

# For UEFI systems:
sudo grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=GRUB

Common GRUB Commands

# List available drives and partitions
ls
# (hd0) (hd0,msdos1) (hd0,msdos2)

# Boot manually
set root=(hd0,msdos1)
linux /vmlinuz-6.2.0-26-generic root=/dev/sda1
initrd /initrd.img-6.2.0-26-generic
boot

Filesystem Repair

# Check filesystem (unmount first!)
sudo umount /dev/sda1

# ext4
sudo fsck.ext4 -f -y /dev/sda1

# XFS
sudo xfs_repair /dev/sda1
# (XFS replay requires a mounted journal first: mount, then repair)
sudo mount /dev/sda1 /mnt 2>/dev/null; sudo umount /mnt
sudo xfs_repair /dev/sda1

# Btrfs (read-only check first)
sudo btrfs check /dev/sda1
sudo btrfs check --repair /dev/sda1  # Only if necessary!

# ZFS
sudo zpool scrub tank
sudo zpool clear tank
sudo zpool export tank; sudo zpool import tank

Superblock Recovery (ext4)

If the primary superblock is corrupted:

# Find backup superblocks
sudo mke2fs -n /dev/sda1

# Use a backup superblock
sudo fsck.ext4 -b 32768 /dev/sda1

Expected output:

$ sudo fsck.ext4 -f -y /dev/sda1
e2fsck 1.46.5 (30-Dec-2021)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sda1: 123456/789012 files (0.1% non-contiguous), 987654/3456789 blocks

Data Recovery

ddrescue — Recover Data from Failing Drives

# Install
sudo apt install gddrescue

# Basic recovery (mapfile tracks progress, resume-safe)
sudo ddrescue -d /dev/sdb /dev/sdc rescue.mapfile

# Retry bad sectors three times
sudo ddrescue -d -r3 /dev/sdb /dev/sdc rescue.mapfile

# Direct disk access, no cache
sudo ddrescue -d -f -A /dev/sdb /dev/sdc rescue.mapfile

Expected output:

GNU ddrescue 1.25
Press Ctrl-C to interrupt
     ipos:   512123 MB, non-trimmed:    0 B,  current rate:  12345 kB/s
     opos:   512123 MB, non-scraped:    0 B,  average rate:  23456 kB/s
non-tried:        0 B,  bad-sector:   4096 B,    error rate:     0 B/s
  rescued:  512123 MB,   busy:        0 B,  finished:  100.00%

Recover Deleted Files

# Install testdisk
sudo apt install testdisk

# Recover partitions
sudo testdisk /dev/sda

# Undelete files (ext4)
sudo extundelete /dev/sda1 --restore-all

# PhotoRec (file carving by signature, works on any filesystem)
sudo photorec /dev/sda1

Root Password Reset

# Method 1: GRUB init=/bin/bash
# At GRUB, edit boot params, add "init=/bin/bash rw" to linux line

# Method 2: Recovery mode with chroot
# Boot live USB, chroot, then:
sudo chroot /mnt
passwd
exit

# Method 3: Single-user mode
# Append "single" to GRUB linux line
# System boots to root shell without password prompt

Live System Diagnostics

# Check SMART status
sudo smartctl -a /dev/sda

# Check RAID status
cat /proc/mdstat

# Check LVM
sudo vgdisplay
sudo lvdisplay

# Memory test
sudo memtest86+  # (requires reboot)

# System log review
journalctl -xb -p emerg --no-pager
journalctl -xb -p alert --no-pager

# Check recent boot failures
journalctl --list-boots
sudo journalctl -b -1 -p err

Common Errors

1. Chroot Fails with "failed to run command" or Shell Not Found

The live system architecture does not match the installed system. Use a 64-bit live USB for 64-bit installations, or a 32-bit one for 32-bit. Check with file /mnt/bin/bash.

2. GRUB Rescue Says "unknown filesystem"

GRUB cannot recognize the filesystem type. This often happens with ZFS, Btrfs with specific features, or LUKS. Use a GRUB module or boot from a live USB and reinstall GRUB with the correct modules.

3. Filesystem Check Finds Many Errors

If fsck finds hundreds of errors, the filesystem or disk is failing. Back up data immediately and replace the drive. Run smartctl -a to check disk health.

4. Root Partition Full — System Fails to Boot

Boot with init=/bin/bash, delete unnecessary files or logs, then boot normally. Prevent with log rotation and monitoring disk usage.

5. ddrescue Stalls on Bad Sectors

A failing drive can take hours per bad sector. Use -A (skip reads that cause errors) to bypass bad areas and come back to them with -r.

6. LVM Volume Groups Missing After Chroot

Run vgscan --mknodes and vgchange -ay inside the chroot. LVM metadata may not activate automatically in a chroot environment.

7. Network Not Working in Rescue Mode

Live environments often do not enable networking by default. Use dhclient or configure the interface manually: ip link set eth0 up && dhclient eth0.

Practice Questions

1. What is the first thing to do when a Linux server does not boot? Boot from a live USB, mount the root filesystem, and check journalctl for boot errors: journalctl -b -1 -p err.

2. How do you reset a forgotten root password without reinstalling? Boot with init=/bin/bash rw in GRUB, then run passwd at the shell.

3. What tool recovers data from a failing disk by tracking progress in a mapfile? ddrescue (gddrescue). The mapfile allows resuming interrupted recoveries.

4. How do you repair a broken GRUB installation from a live USB? Mount the root partition, chroot into it, then run grub-install /dev/sda and update-grub.

5. What command checks an ext4 filesystem for errors? sudo fsck.ext4 -f /dev/sda1 (unmount the filesystem first).

Challenge: Create a realistic rescue scenario: install a VM, corrupt its GRUB by zeroing the first 512 bytes (dd if=/dev/zero of=/dev/sda bs=512 count=1), boot from a live ISO, chroot into the system, reinstall GRUB, and boot the VM successfully. Document every step and the expected error messages at each stage.

What's Next

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro. Updated 2026-06-24.