Prometheus's Data Model: Metric Name + Labels

Scenario: A team wants to know the error rate of their API, broken down by which endpoint and which HTTP status code caused it - without writing a separate metric for every single endpoint/status combination by hand.

New words, in plain English

• Metric name - the human-readable name of what is being measured - http_requests_total, cpu_usage_seconds.
• Label - a key/value tag attached to a metric that describes one dimension of it - which endpoint, which status code, which server.
• Label value - the specific value of one label on one particular measurement - for the label status, a label value might be "200" or "500".
• Time series - one unique combination of a metric name and a full set of label values, tracked over time as its own line.
• Cardinality - how many distinct time series a metric produces in total, across all its label combinations - covered in depth in Phase 5.

Every Prometheus metric is really two things bolted together: a name that says what is being measured, and a set of labels that say which specific instance of that measurement this is. http_requests_total is the name; {method="GET", status="500", endpoint="/checkout"} are the labels attached to one particular reading of it.

This matters because it means you do not need a different metric name for every variation you might want to slice by. Instead of inventing http_requests_total_get_200, http_requests_total_get_500, http_requests_total_post_200 and so on forever, you keep one metric name and let labels carry the variation. Each unique combination of metric name plus label values is its own time series - its own independent line that Prometheus tracks over time.

This is also what makes Prometheus's query language, PromQL, so flexible: you can ask for the whole family of time series under one metric name, or narrow it down to an exact combination of labels, using the same underlying data.

Analogy: Think of a metric name as a filing cabinet's label - "Sales Receipts" - and each label as one drawer inside it. One drawer might be region="east", product="widgets", another region="west", product="gadgets". You still have one cabinet (one metric name) but you can pull exactly the drawer you need, or pull every drawer in the cabinet, without needing a separate cabinet for every possible region/product combination.

A worked example

# ONE metric name, MANY time series underneath it - each unique label
# combination is tracked separately by Prometheus.
http_requests_total{method="GET",  status="200", endpoint="/checkout"} 91422
http_requests_total{method="GET",  status="500", endpoint="/checkout"} 37
http_requests_total{method="POST", status="200", endpoint="/checkout"} 8103
http_requests_total{method="GET",  status="200", endpoint="/login"}    50210

# A simple PromQL query narrows this down using labels:
http_requests_total{endpoint="/checkout", status="500"}

Choosing good labels is a real design skill, not an afterthought. A label should describe something with a small, bounded number of possible values - an HTTP method, a status code, a region, a service name. What a label should almost never hold is something with effectively unlimited possible values, like a raw user ID or a full request URL with query parameters - doing that quietly explodes the number of time series Prometheus has to track, which is exactly the cardinality problem this track returns to in Phase 5.

Tip: A useful mental check before adding a new label: "could this value be one of thousands or millions of different things?" If yes, it probably belongs in a log line, not a Prometheus label.

Goal: Put this to work in the promgraf-configure-scrape-target lab. Open /labs/prometheus-grafana, pick promgraf-configure-scrape-target, and fix the real broken monitoring stack - a genuine Prometheus server scraping real targets, and a genuine Grafana instance querying it.