Pull vs. Push: Two Ways to Collect a Metric

Scenario: A team is choosing a monitoring tool. One engineer says every application should call out to a central server whenever a metric changes. Another says the opposite: a central server should visit each application and ask it for its current numbers.

New words, in plain English

• Push model - each application actively sends its metrics to a central collector, usually over the network, whenever it wants to.
• Pull model - a central collector reaches out to each application on a schedule and asks "what are your current metric values?".
• Scrape - Prometheus's word for one pull: visiting a target and reading its current metrics.
• Target - one thing Prometheus scrapes - usually one running application or service, identified by a network address.
• Endpoint - a specific URL a service exposes so something else can reach it - here, the URL Prometheus scrapes for metrics, conventionally /metrics.

There are two fundamentally different ways to get a metric from an application into a monitoring system.

In a push model, the application itself is responsible for reaching out and sending its numbers somewhere - like a factory phoning head office every hour to report production figures. In a pull model, the monitoring system does the opposite: it visits every application on a fixed schedule and asks, "what are your numbers right now?" - like an inspector who walks the factory floor at 9 AM sharp and reads every gauge in turn.

Prometheus deliberately chose pull. A Prometheus server is given a list of targets (addresses of applications to monitor) and, on a fixed interval, sends an HTTP request to each one's /metrics endpoint. Each application does not need to know anything about where Prometheus lives, how many Prometheus servers exist, or what to do if the network to head office is briefly down - it just needs to expose its current numbers whenever asked, which is a far simpler contract than "remember to actively phone home reliably".

Pull has a very practical side effect that matters for day-to-day operations: because Prometheus is the one initiating contact on a known schedule, a target that stops responding is itself useful information - Prometheus can tell you "I have not been able to reach this application for the last two minutes", which is a distinct and valuable signal from "this application reported zero errors".

Analogy: A push model is a student who is supposed to email their teacher a progress report whenever they feel like it - if the student forgets, or their email is broken, the teacher may never notice. A pull model is the teacher walking down the row of desks every ten minutes and looking at exactly what each student has written so far. The teacher never has to wonder if a student forgot to report in - either their desk has work on it, or the teacher can see the desk is empty.

A worked example

# What Prometheus is told about ONE target to pull from.
# Every 15 seconds, Prometheus will HTTP GET this address's /metrics endpoint.
scrape_configs:
  - job_name: "checkout-service"
    scrape_interval: 15s
    static_configs:
      - targets: ["checkout-service:9100"]

# What Prometheus actually receives back from that endpoint - plain text,
# one metric per line, that the application generated itself:
http_requests_total{method="GET",status="200"} 18234
http_requests_total{method="GET",status="500"} 12
process_resident_memory_bytes 41943040

Pull is not automatically superior in every situation - short-lived jobs that finish and exit before Prometheus ever gets a chance to scrape them (a batch job that runs for four seconds) need a workaround, usually a small helper called a Pushgateway that the job pushes its final numbers to, which Prometheus then pulls from in the normal way. But for the overwhelmingly common case - long-running services like web servers and databases - pull keeps the application simple and gives you free, reliable "is this thing even alive?" information for no extra work.

Warning: A target that Prometheus cannot reach does not mean "zero traffic" - it means "unknown", and the two look identical on a graph unless you specifically check the target's up/down status. Confusing "no data" with "nothing happened" is one of the most common monitoring mistakes.

Goal: Put this to work in the promgraf-configure-scrape-target lab. Open /labs/prometheus-grafana, pick promgraf-configure-scrape-target, and fix the real broken monitoring stack - a genuine Prometheus server scraping real targets, and a genuine Grafana instance querying it.