Stop the Alert Fatigue
An alerting rule with no sustained-duration check pages on every tiny fluctuation instead of real problems.
A intermediate Prometheus & Grafana challenge worth 10 points. Solve it hands-on in a real Prometheus & Grafana environment in your browser - no local setup, no fake shells.
The Challenge
Scenario: On-call is drowning. An alerting rule fires the instant a metric crosses its threshold for even a single scrape, so every brief, self-resolving blip pages someone - a real documented failure mode where unfiltered notifications on overly-broad conditions train people to ignore every page, including the real ones.
Your tasks:
- Find the flapping alerting rule in the Prometheus rules file.
- Add a
for:duration so the condition must hold continuously before it fires, instead of on a single scrape. - Reload Prometheus and confirm via
/api/v1/rulesthat the rule now requires sustained breach, not an instant blip.
Commands to learn: Prometheus alerting rule for: clause, curl localhost:9090/api/v1/rules.
Run check when done.