The Cardinality Bomb

A metric labeled with a raw per-request ID is generating a new time series on almost every scrape.

A advanced Prometheus & Grafana challenge worth 20 points. Solve it hands-on in a real Prometheus & Grafana environment in your browser - no local setup, no fake shells.

The Challenge

Scenario: Based on a real postmortem: a request counter labeled with a raw user_id generated over 5 million distinct time series in 90 days and OOM-killed Prometheus. This lab reproduces the same pattern at a survivable scale - a demo app is exposing a metric with an ever-changing label value, and cardinality is climbing scrape by scrape.

Your tasks:

1. Inspect the demo app's exposed metrics and identify the label generating a new value almost every scrape.
2. Add a metric_relabel_configs rule to that scrape job to drop the offending label (or the whole metric).
3. Confirm new high-cardinality series stop being created going forward.

Commands to learn: metric_relabel_configs, action: labeldrop / action: drop, inspecting series counts via the TSDB.

Run check when done.