Alertmanager: Routing, Grouping and Silencing

Scenario: Twelve alerting rules all fire within the same thirty seconds because one shared database went down, and twelve separate, near-identical pages land in an on-call engineer's phone at once.

New words, in plain English

• Alertmanager - a separate component that receives firing alerts from Prometheus and decides what to actually do with them - who to notify, how, and when.
• Routing - Alertmanager's rules for deciding which team or notification channel a given firing alert should go to, based on its labels.
• Grouping - combining multiple related firing alerts into a single notification, instead of sending one notification per alert.
• Silencing - deliberately muting notifications for a specific alert or set of alerts for a chosen time window - typically during planned maintenance.

Prometheus itself only evaluates rules and decides whether an alert is Firing - it does not send Slack messages, emails or phone calls on its own. That job belongs to a separate component: Alertmanager. Prometheus sends every firing alert to Alertmanager, and Alertmanager is what actually decides where it goes and how it is presented to a human.

Three jobs make Alertmanager worth having as its own component rather than folding this logic into Prometheus itself. Routing looks at an alert's labels (like severity or team) and decides which notification channel or team it belongs to - a severity: critical team: database alert might go to a database team's pager, while a severity: warning alert from the same rule group might just post to a Slack channel. Grouping solves exactly the twelve-simultaneous-pages problem: instead of one notification per alert, Alertmanager can combine related alerts that fired together into a single notification - "12 checkout endpoints are erroring, likely a shared cause" - which is both less noisy and more informative than twelve separate pings. Silencing lets someone deliberately mute notifications for a known issue - planned maintenance on a database, for instance - for a defined time window, so real work does not generate false alarm noise the whole team already knows about.

This track's labs focus on writing correct Prometheus alerting rules, since that is where most real alert-fatigue and alert-fatigue-adjacent bugs originate. Alertmanager's routing/grouping/silencing concepts are introduced here because they are how a real production alerting pipeline continues past "Prometheus decided this is Firing" - useful context even without a full Alertmanager lab setup in this first pass of the track.

Analogy: If Prometheus is the smoke detector deciding "there is definitely a sustained fire," Alertmanager is the building's fire-alarm control panel: it decides which floor's speaker to sound the alarm on (routing), it merges twelve detectors on the same floor going off at once into one alarm event instead of twelve separate sirens (grouping), and it is the switch the maintenance crew flips to say "we know, we're testing the fire door on floor 3, don't call anyone" (silencing).

A worked example

# Conceptual flow - NOT a full Alertmanager config, just the shape of it.
#
#   Prometheus                  Alertmanager                  Human
#   -----------                 ------------                  -----
#   rule fires  ------------->  receives alert
#   (HighErrorRate,                  |
#    severity=critical)         ROUTE by labels
#                                (severity=critical -> pager)
#                                     |
#                                GROUP with any other
#                                alerts that fired together
#                                     |
#                                (unless SILENCED for
#                                 a known maintenance window)
#                                     |
#                                send ONE notification ----> on-call engineer

A practical rule of thumb for labels on alerting rules, given Alertmanager exists downstream: always include a severity label, and be deliberate about what values it can take (critical, warning, info is a common minimal set) since Alertmanager's routing typically keys off exactly that label to decide urgency and destination.

Tip: When designing a new alerting rule, ask not just "what expr fires this?" but "once this fires, who actually needs to know, how urgently, and through what channel?" - that second question is Alertmanager's whole job, and skipping it is how teams end up with alerts nobody is actually routed to see.

Goal: Put this to work in the promgraf-silent-target-down lab. Open /labs/prometheus-grafana, pick promgraf-silent-target-down, and fix the real broken monitoring stack - a genuine Prometheus server scraping real targets, and a genuine Grafana instance querying it.