Prometheus's Architecture: Scraper, TSDB, HTTP API

Scenario: A new hire is told "Prometheus is running, it's collecting our metrics" and asked to go debug why a graph looks wrong. They open the Prometheus server and have no idea which of its internal parts to even look at.

New words, in plain English

Prometheus server - the single binary that does everything: scrapes targets, stores the data, evaluates queries and rules, and answers HTTP requests.
TSDB (time-series database) - Prometheus's own built-in storage engine, purpose-built for storing huge numbers of timestamped numbers efficiently.
HTTP API - the network interface Prometheus exposes so other tools - including Grafana and the promtool/web UI - can ask it questions.
Scrape loop - the repeating cycle where Prometheus works through its target list and pulls fresh metrics from each one.
Retention - how long Prometheus keeps old data before deleting it, covered in depth in Phase 5.

Unlike some monitoring systems that split collection, storage and querying across several separate services you have to wire together, Prometheus deliberately ships as one binary with three jobs built in.

First, the scrape loop works through the list of targets from its configuration on a repeating schedule, pulling the current metrics text from each one's /metrics endpoint. Second, everything it collects is written into TSDB, Prometheus's own on-disk time-series database, optimized specifically for the pattern "lots of numbers, tagged with timestamps and labels, that need writing constantly and querying by time range." Third, an HTTP API sits on top of that data so anything - the built-in web UI, promtool, or an external tool like Grafana - can send it a PromQL query and get an answer back.

This single-binary design is a deliberate simplicity trade-off: a single Prometheus server is easy to reason about and run, at the cost of needing extra tooling (like federation or remote-write to a longer-term store) once you outgrow what one server's local disk can hold.

Analogy: A single Prometheus server is like a small town's one combined post office, sorting depot and records office rolled into one building. The mail carrier (scrape loop) goes out on their route and collects letters (metrics) on a schedule. Everything collected gets filed in the building's own archive room (TSDB). And the front desk (HTTP API) is where anyone - a resident, or another office across town (Grafana) - comes to ask "what mail did house number 12 receive last Tuesday?"

A worked example

# The three jobs Prometheus does with itself as an example target -
# Prometheus scrapes its OWN /metrics endpoint by default, so you can
# see this loop happening from the moment it starts.

#            scrape loop              TSDB                HTTP API
#          (every 15s, pull)     (writes to disk)      (answers queries)
#                |                     |                     |
#   targets ---> GET /metrics -----> store ------------> promtool / UI / Grafana
#   (self,      (reads current       (with a               ask PromQL, get
#    checkout,   values as plain      timestamp             back a time series
#    db, ...)    text)                attached)

Because it is one binary, restarting the Prometheus server means both the scraping and the query-answering stop briefly together - there is no separate "storage service" that keeps running independently. In production this single-binary simplicity is usually paired with running more than one Prometheus instance for redundancy, but for learning and for small setups, one server doing all three jobs is the whole picture.

Tip: Prometheus's own built-in web UI (usually on port 9090) lets you run PromQL queries directly and see the raw scraped metrics with no Grafana involved at all - a good first stop when debugging whether a problem is in Prometheus or in a dashboard built on top of it.

Goal: Put this to work in the promgraf-configure-scrape-target lab. Open /labs/prometheus-grafana, pick promgraf-configure-scrape-target, and fix the real broken monitoring stack - a genuine Prometheus server scraping real targets, and a genuine Grafana instance querying it.