Writing a scrape_config

Scenario: A checkout service exposes metrics on port 9100, but Prometheus's dashboard shows no data for it at all - the service was simply never added to the list of things Prometheus knows to scrape.

New words, in plain English

• scrape_configs - the section of Prometheus's configuration file that lists everything it should scrape, and how.
• job_name - a human-readable label naming one group of targets being scraped for the same purpose - all the replicas of one service, for example.
• static_configs - the simplest way to list targets: writing their addresses directly in the configuration file.
• scrape_interval - how often Prometheus pulls fresh metrics from a target - 15s means once every fifteen seconds.
• prometheus.yml - the conventional filename for Prometheus's main configuration file.

Prometheus only scrapes what it is explicitly told to scrape - there is no automatic discovery of "every service on the network" unless you deliberately configure a discovery mechanism (briefly covered in Phase 5). The plain, static way to tell it is a scrape_config entry in prometheus.yml.

Each entry needs a job_name - a label that groups related targets together and gets automatically attached to every metric scraped under it, so you can later tell "this reading came from the checkout-service job" apart from "this reading came from the database job." It needs static_configs, which lists the actual network addresses (host:port) to scrape. And it typically sets scrape_interval to control how often that happens - if omitted, Prometheus falls back to its own global default.

After editing the file, Prometheus needs to either be restarted or sent a reload signal before the new configuration takes effect - simply saving the file does nothing on its own.

Analogy: prometheus.yml's scrape_configs section is a mail carrier's route sheet. Each job_name is one named route ("downtown apartments", "industrial park"), static_configs is the literal list of addresses on that route, and scrape_interval is how often the carrier walks that route in a day. A house that never made it onto any route sheet simply never gets visited, no matter how much mail is piling up on its porch.

A worked example

# prometheus.yml
global:
  scrape_interval: 15s   # the default, used when a job doesn't set its own

scrape_configs:
  - job_name: "prometheus"        # Prometheus scraping itself
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "checkout-service"
    scrape_interval: 10s          # this job overrides the global default
    static_configs:
      - targets: ["checkout-service:9100"]

  - job_name: "database-exporter"
    static_configs:
      - targets: ["db-exporter:9187"]

A target that is misspelled, on the wrong port, or simply missing from scrape_configs entirely produces no error message anywhere obvious - it just silently never appears in Prometheus's data. The Prometheus web UI's Targets page is the first place to check: it lists every configured target along with its current UP or DOWN state and the exact error if a scrape is failing, which is usually far faster than guessing from a blank Grafana panel.

Warning: Editing prometheus.yml alone changes nothing until Prometheus reloads it - either a full restart, or (if enabled) a reload signal/endpoint. A perfectly correct config that was never reloaded looks identical, from a dashboard, to a wrong config.

Goal: Put this to work in the promgraf-configure-scrape-target lab. Open /labs/prometheus-grafana, pick promgraf-configure-scrape-target, and fix the real broken monitoring stack - a genuine Prometheus server scraping real targets, and a genuine Grafana instance querying it.