Retention: Why Unbounded History Fills a Disk

Scenario: A Prometheus server that has been running quietly for eight months suddenly starts refusing to start after a routine restart - its data directory has silently grown to fill the entire disk.

New words, in plain English

• Retention period - how long Prometheus keeps stored data before automatically deleting the oldest of it - configured with a flag such as --storage.tsdb.retention.time.
• Disk usage growth - the fact that every scrape, from every target, writes new data to disk forever, so storage use only ever grows unless something is deleted.

Every single scrape Prometheus performs writes new samples to TSDB. With no retention limit configured, that data simply accumulates forever - a year of history from a busy fleet of targets can be a genuinely enormous amount of data, and eventually the disk holding it fills up completely.

A Prometheus server that runs out of disk space does not fail gracefully in a way that is easy to notice from a dashboard - it can stop being able to write new data, or fail to restart at all, which is exactly the kind of self-inflicted outage that is entirely avoidable with a sensible retention period set from day one.

Retention is the deliberate trade-off: keep enough history to be useful ("what did this look like a month ago, before we made that change?") without keeping so much that it becomes a liability. Fifteen days is Prometheus's own long-standing default for a reason - useful for recent incident review without unbounded growth - though many teams tune it higher or lower for their own needs, or pair a shorter local retention with a separate long-term remote storage system for anything that truly needs to be kept for months or years.

Analogy: A single Prometheus server with no retention limit is like a filing cabinet that a company decided would simply never be emptied - keeping every receipt, every memo, every draft, forever, on the assumption that space is not a real limit. Eventually the room the cabinet sits in runs out of floor space entirely, and the failure mode is not "a little cramped" - it is "no more filing is possible at all."

A worked example

# Retention is set as a Prometheus startup flag, not in prometheus.yml.
# This keeps 15 days of local history - deleting anything older automatically.
prometheus --config.file=prometheus.yml --storage.tsdb.retention.time=15d

# Checking how much disk the TSDB is actually using is a routine health check:
du -sh /prometheus/data

Retention interacts directly with cardinality (the next section): the total disk used is roughly proportional to both how long you keep data AND how many distinct time series exist. A team that fixes a cardinality explosion but never revisits retention, or vice versa, has only solved half of a disk-growth problem.

Warning: A Prometheus server silently approaching its disk limit typically shows no obvious symptom until it is already critical - by the time writes actually start failing, you are already in an incident, not planning ahead of one. Monitoring Prometheus's own disk usage as a metric, checked well before it is full, is a standard defensive habit.

Goal: Put this to work in the promgraf-silent-target-down lab. Open /labs/prometheus-grafana, pick promgraf-silent-target-down, and fix the real broken monitoring stack - a genuine Prometheus server scraping real targets, and a genuine Grafana instance querying it.