Alerting Rules: expr, for, labels and annotations

Scenario: A team has a beautiful dashboard showing error rate, but every incident is still discovered by a customer complaining first - nobody was watching the graph at 3 AM when it spiked.

New words, in plain English

• Alerting rule - a PromQL condition Prometheus continuously evaluates; when it becomes true, an alert fires.
• expr - the alerting rule's field holding the actual PromQL expression to evaluate - the condition being watched.
• for - how long a condition must stay continuously true before the alert actually fires, rather than firing the instant it becomes true once.
• labels (on an alert) - extra tags attached to a firing alert - such as severity - used for routing it to the right place.
• annotations - human-readable text attached to a firing alert - a summary and description meant to be read by a person during an incident.
• Pending vs. Firing - an alert whose condition just became true enters Pending; it only becomes Firing once it has stayed true for the full for duration.

A dashboard only helps if a human is actively looking at it. Alerting rules flip that around: instead of a human watching a graph, Prometheus itself continuously evaluates a condition and proactively notifies someone the moment it becomes true.

An alerting rule has four key parts. The expr is the actual PromQL condition being watched - for example, "error rate is above 5%." The for duration says how long that condition has to stay continuously true before the alert is treated as real, rather than firing on the very first instant it becomes true. labels attach metadata to the firing alert itself - most importantly severity, which downstream tooling uses to decide how urgently to notify someone. annotations hold the human-readable text - a summary and description - that actually gets read by the on-call engineer when the notification arrives.

When a rule's expr first becomes true, the alert enters a Pending state, not Firing yet. Only once it has remained continuously true for the entire for duration does it transition to Firing and actually trigger a notification. If the condition stops being true before for elapses, the alert quietly resets back to inactive - no notification is ever sent.

Analogy: A smoke detector that screams the instant a single wisp of smoke drifts past - from toast, not fire - trains everyone in the house to ignore it. A smoke detector that waits until smoke has been continuously present for a short sustained period (its own version of for) before sounding is the one people still trust and react to after a year of daily cooking.

A worked example

groups:
  - name: checkout-alerts
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{job="checkout-service", status="500"}[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Checkout service error rate is high"
          description: "More than 5% of checkout requests have failed for the last 5 minutes."

A rule with no for at all (or for: 0s) fires the instant the expression is true even once, at the very next scrape - which is exactly what turns a single transient blip, like one slow database query, into a 3 AM page for something that was already fine thirty seconds later. Choosing a for duration is a genuine judgment call: too short and you get noise; too long and a real, sustained outage takes longer than necessary to notify anyone about. There is no universal right number - it depends on how quickly the underlying condition is expected to naturally resolve versus how urgently a human needs to be told.

Warning: This is the direct cause of alert fatigue: a team that gets paged for every brief, self-resolving blip starts distrusting - and eventually ignoring - every alert, including the real ones. A for duration matched to how the underlying system actually behaves is the single most effective fix.

Goal: Put this to work in the promgraf-alert-fatigue-flapping lab. Open /labs/prometheus-grafana, pick promgraf-alert-fatigue-flapping, and fix the real broken monitoring stack - a genuine Prometheus server scraping real targets, and a genuine Grafana instance querying it.