Service Discovery, and What Goes Wrong in Production

Scenario: A platform team running 200 auto-scaling containers realizes that hand-editing a static list of targets every time a container starts or stops is not sustainable - by the time anyone edits the file, the list is already out of date.

New words, in plain English

• Service discovery - letting Prometheus automatically find its list of targets from a live source - like a cloud API or a container orchestrator - instead of a hand-maintained static list.
• Dynamic target list - a set of scrape targets that updates itself automatically as real infrastructure is created and destroyed, rather than staying fixed until someone edits a config file.

Everything in Phase 2's scrape_config used static_configs - a fixed, hand-written list of addresses. That works well for a small, stable set of services, but it breaks down at real scale: in an environment where containers or virtual machines are constantly being created and destroyed by an orchestrator or auto-scaler, a static list is stale almost as soon as it is written, either missing brand-new targets entirely or still trying (and failing) to scrape ones that no longer exist.

Service discovery is Prometheus's answer: instead of a static list, Prometheus is configured to ask a live source - a cloud provider's API, a container orchestrator, a DNS record - "what targets currently exist?", and it automatically updates its own target list as the answer changes over time, with no human needing to edit prometheus.yml every time infrastructure changes. This track's labs use static configuration throughout, since it is the right starting point for actually understanding what a target and a scrape are - service discovery is a direct, layered extension of that same underlying model once an estate outgrows a hand-maintained list.

Bringing this whole phase together: production monitoring failures are rarely about Prometheus or Grafana being fundamentally broken software - they are almost always one of the specific, learnable failure patterns this track has now covered. A target silently never gets added to scrape_configs (Phase 2). A dashboard shows nothing because no datasource was ever wired up (Phase 3). An alert with no for duration pages someone for a blip that resolved itself thirty seconds later (Phase 4). A disk fills up because nobody set a retention period, or because one high-cardinality label quietly multiplied the number of time series being tracked (this phase). Each of these is exactly the kind of realistic, broken-on-purpose scenario the labs in this track are built around.

Analogy: A static target list is a paper phone book printed once a year - accurate the day it was printed, and increasingly wrong every day after as people move and change numbers. Service discovery is a live directory service that is always queried fresh - it never goes stale, because it is never "printed" in the first place; it is asked, and answered, at the moment it is needed.

A worked example

# What static configuration looks like (what this track's labs use):
scrape_configs:
  - job_name: "checkout-service"
    static_configs:
      - targets: ["checkout-service:9100"]

# What dynamic service discovery looks like conceptually (not run in
# this track's labs, but the natural next step at larger scale):
scrape_configs:
  - job_name: "checkout-service"
    kubernetes_sd_configs:      # "ask the Kubernetes API what pods exist"
      - role: pod
    # Prometheus now automatically tracks pods as they are created
    # and destroyed, with no manual editing of the target list.

The five real, corporate-incident-style labs in this track are deliberately built around exactly the failure patterns this whole curriculum has walked through: a target that was never correctly configured for scraping (promgraf-configure-scrape-target), a Grafana instance with no working connection to its data (promgraf-grafana-datasource-setup), a dashboard that needed to be brought in and wired up correctly (promgraf-import-default-dashboard), an alerting rule missing the for duration that would have prevented alert fatigue from flapping (promgraf-alert-fatigue-flapping), and a target quietly going down with nobody noticing because nothing was watching for it (promgraf-silent-target-down). Every concept in this track exists to make those five scenarios - and the real incidents they are modeled on - solvable by understanding, not by guessing.

Tip: When a production monitoring system misbehaves, work backwards through this track's phases in order: is the target even being scraped (Phase 2)? Is Grafana even connected to the right data (Phase 3)? Is the alert rule shaped correctly (Phase 4)? Is disk or cardinality the real root cause (this phase)? Most real incidents land cleanly on one of these four questions.

Goal: Put this to work in the promgraf-silent-target-down lab. Open /labs/prometheus-grafana, pick promgraf-silent-target-down, and fix the real broken monitoring stack - a genuine Prometheus server scraping real targets, and a genuine Grafana instance querying it.