Disk Full Before Morning
Prometheus has no retention.size flag against a fixed disk budget and is heading for a crash-loop.
A advanced Prometheus & Grafana challenge worth 20 points. Solve it hands-on in a real Prometheus & Grafana environment in your browser - no local setup, no fake shells.
The Challenge
Scenario: Prometheus's data directory sits on a disk with a fixed, small budget, but nobody ever set --storage.tsdb.retention.size. Left alone, it will fill the disk overnight and crash-loop by morning.
Your tasks:
- Check the current disk budget available to Prometheus's data directory.
- Set
--storage.tsdb.retention.sizeto roughly 80-85% of that budget, per Prometheus's own guidance, leaving headroom for in-progress compaction. - Restart Prometheus with the new flag and confirm it's actually applied.
Commands to learn: --storage.tsdb.retention.size, df, checking a running process's flags.
Run check when done.