The Migration That Broke Every Dashboard

A relabel_configs change during a migration silently renamed a label, and a saved dashboard query now returns nothing.

A advanced Prometheus & Grafana challenge worth 20 points. Solve it hands-on in a real Prometheus & Grafana environment in your browser - no local setup, no fake shells.

The Challenge

Scenario: During a recent migration, someone changed a scrape job's relabel_configs to rename a label - and never told anyone. A saved Grafana dashboard that queries by the old label name now returns nothing, and looks like an outage even though the underlying metrics are fine.

Your tasks:

1. Compare the dashboard's saved query against the label the metric actually carries now.
2. Find the relabel_configs entry responsible for the rename in prometheus.yml.
3. Fix or remove the relabel rule so the metric keeps the label name/value existing consumers rely on.
4. Restart Prometheus and re-run the saved query to confirm it returns data again.

Commands to learn: relabel_configs vs metric_relabel_configs, PromQL label matching.

Run check when done.