Cardinality: The Other Way to Fill a Disk

Scenario: A well-intentioned engineer adds a user_id label to a metric so support can look up any individual customer's request count. Within a week, Prometheus's memory usage triples and query performance across the entire dashboard suite degrades.

New words, in plain English

Recall from Phase 1 that a time series is one unique combination of metric name and label values. Cardinality is simply the count of how many of those distinct combinations exist. A metric with two labels, each taking one of three possible values, has at most nine distinct time series. That is a small, bounded, entirely manageable number.

The problem is what happens when a label's possible values are not small and bounded, but effectively unlimited - a raw user ID, a full request URL including query parameters, a UUID generated fresh for every single request. Adding a label like that to a metric does not add "a few more time series" - it can create a new, permanent time series for every single distinct value that has EVER appeared, and Prometheus keeps tracking every one of them, forever (or until retention expires them), each consuming its own memory and disk.

This is exactly why Phase 1 warned against putting anything with unlimited possible values into a label: a cardinality explosion is one of the single most common ways a Prometheus server that was running fine for months suddenly starts using dramatically more memory and disk, and starts answering queries dramatically slower, seemingly overnight - usually traced back to one recently-added label.

Analogy: A label is like a filing cabinet drawer, as Phase 1 put it - grouping related documents together, like "documents about the East region." A reasonable label creates a handful of drawers: East, West, North, South. A high-cardinality label is like deciding to give every single piece of paper its own individually labeled drawer, instead of grouping papers into a manageable number of drawers - suddenly you need a warehouse of millions of one-sheet drawers instead of a small filing cabinet, and finding anything, or even just storing it all, becomes dramatically harder and more expensive.

A worked example

# LOW cardinality - status has a small, fixed set of realistic values.
# A handful of time series, easy to store and query forever.
http_requests_total{status="200"}
http_requests_total{status="404"}
http_requests_total{status="500"}

# HIGH cardinality - user_id has effectively unlimited possible values.
# This creates a BRAND NEW, permanent time series for every user who has
# EVER made a request - potentially millions, and growing forever.
http_requests_total{status="200", user_id="883211"}
http_requests_total{status="200", user_id="883212"}
http_requests_total{status="200", user_id="883213"}
# ... one new time series per distinct user, forever

The fix for "I need to look up one specific user's activity" is almost never "put the user ID in a Prometheus label" - that specific, high-detail, per-request question is exactly what logs (from Phase 1) are the right tool for. Prometheus and its labels are for bounded dimensions worth graphing and alerting on in aggregate; anything with unlimited distinct values belongs in a log line instead, where it can be searched without permanently bloating every metric query across the whole system.

Tip: Before adding any new label to a metric, ask: "if I listed every value this label could ever realistically take, would that list be short and stable, or open-ended and ever-growing?" Short and stable is safe. Open-ended is a cardinality explosion waiting to happen.
Goal: Put this to work in the promgraf-silent-target-down lab. Open /labs/prometheus-grafana, pick promgraf-silent-target-down, and fix the real broken monitoring stack - a genuine Prometheus server scraping real targets, and a genuine Grafana instance querying it.