Disk Full Production Incident: Finding Hidden Space Usage

The Alert

6:22 AM. Root filesystem at 100%. The app throws 500s — it cannot write session files.

---

Step 1: Find Where Space Is Used

df -h
du -sh /* 2>/dev/null | sort -rh | head -10
du -sh /var/* 2>/dev/null | sort -rh | head -10
du -sh /var/lib/* 2>/dev/null | sort -rh | head -5

Result: /var/lib/docker = 35 GB. But df shows 80 GB used vs 72 GB from du — an 8 GB gap.

---

Step 2: The du/df Gap — Deleted Files Still Held Open

lsof | grep deleted | sort -k7 -rn | head -10
nginx  1234  root  5w  REG  8,1  7516192768  deleted  /var/log/nginx/access.log

A 7.5 GB log file was rotated (deleted) but nginx still holds the file descriptor. Blocks not freed until nginx closes it.

systemctl restart nginx    # releases the file descriptor → immediate 7.5 GB freed

---

Step 3: Inode Exhaustion

df -i
Filesystem  Inodes  IUsed  IFree  IUse%
/dev/sda1   5M      5M     0      100%
find / -xdev -printf '%h
' 2>/dev/null | sort | uniq -c | sort -rn | head -5
3821045  /var/spool/postfix/maildrop

3.8 million queued mail files from a misconfigured mailer.

postsuper -d ALL

Runs 10 minutes but frees all 3.8 M inodes.

---

Step 4: Docker Layer Cleanup

docker system df

18 GB of reclaimable image layers (stopped containers, old versions, dangling layers).

docker system prune -a --volumes

---

Step 5: Fix Log Rotation

The rotation was running but nginx's signal changed after an upgrade. The rotated file was deleted but nginx kept writing to it.

/var/log/nginx/*.log {
    daily
    rotate 14
    compress
    maxsize 500M
    missingok
    notifempty
    postrotate
        /bin/kill -USR1 $(cat /run/nginx.pid 2>/dev/null) 2>/dev/null || true
    endscript
}

---

Post-Mortem: Four Separate Causes

  1. Log rotation broken for 6 months → 7.5 GB held open
  2. Postfix queue never drained → 3.8 M inodes
  3. Docker images never pruned → 18 GB dangling layers
  4. No disk alerting until 100% (should alert at 80%)

Practice in the ShellGenius Linux Labs — the ghost-disk-usage and filesystem-corruption-recovery challenges cover the exact tools used here.