Stop the Alert Fatigue

An alerting rule with no sustained-duration check pages on every tiny fluctuation instead of real problems.

A intermediate Prometheus & Grafana challenge worth 10 points. Solve it hands-on in a real Prometheus & Grafana environment in your browser - no local setup, no fake shells.

The Challenge

Scenario: On-call is drowning. An alerting rule fires the instant a metric crosses its threshold for even a single scrape, so every brief, self-resolving blip pages someone - a real documented failure mode where unfiltered notifications on overly-broad conditions train people to ignore every page, including the real ones.

Your tasks:

  1. Find the flapping alerting rule in the Prometheus rules file.
  2. Add a for: duration so the condition must hold continuously before it fires, instead of on a single scrape.
  3. Reload Prometheus and confirm via /api/v1/rules that the rule now requires sustained breach, not an instant blip.

Commands to learn: Prometheus alerting rule for: clause, curl localhost:9090/api/v1/rules.

Run check when done.