Metrics, Logs and Traces: Three Different Questions

Scenario: A checkout service starts failing intermittently at 2 AM. One engineer has a dashboard that shows error rate rising. Another has to SSH into a server and grep through a log file by hand. A third has no idea which of the twelve backend calls in a single checkout request is the slow one.

New words, in plain English

• Observability - the general ability to ask questions about a running system and get answers, without having to guess or add new code first.
• Metric - a single number that changes over time - how many requests per second, how much memory is used, how many errors happened.
• Log - a timestamped line of text a program writes when something happens - "user 4021 logged in", "connection refused".
• Trace - the recorded path of one single request as it travels through multiple services, showing how long each step took.
• Time series - one metric's value recorded again and again over time - like a single row in a spreadsheet where each column is a moment in time.

These three tools answer three different questions, and none of them replaces the other two.

Metrics answer "how much, and how is it trending?" - CPU usage over the last hour, requests per second, how many orders failed in the last five minutes. A metric is cheap to store (it is just numbers) and cheap to query even across months of history, which is exactly why it is the right tool for dashboards and alerts - things that need to be checked constantly and stay fast at scale.

Logs answer "what exactly happened, in detail, at this one moment?" - the full text of an error message, which user triggered it, what request ID it had. Logs are rich but expensive: a busy service can produce gigabytes of log text a day, and searching through it is slower than reading a number off a graph.

Traces answer "where did the time go, across every service this one request touched?" - useful once you already know something is slow and need to find which of many services caused it.

A healthy monitoring setup uses all three together: a metric's graph tells you something is wrong right now, a trace narrows down where, and a log tells you exactly what happened there. This track is about the first of those three - metrics - and the two tools built specifically for them: Prometheus (collects and stores metrics) and Grafana (turns them into dashboards a human can actually read).

Analogy: Think of a hospital. A patient's heart-rate monitor beeping a number every second is a metric - cheap, constant, and exactly what an alarm should watch. The nurse's detailed handwritten notes about a specific incident are logs - rich, but you would never want an alarm scanning paragraphs of prose in real time. And the full record of everywhere the patient was moved during one emergency - the exact path from ward to X-ray to surgery - is a trace: only useful once you already know something went wrong and need to see the whole journey.

A worked example

# Same underlying event, expressed three different ways.

# METRIC - a single number, sampled every 15 seconds
http_requests_total{status="500"} 42

# LOG - one detailed text line for one specific failure
2026-09-09T02:14:03Z ERROR checkout: payment gateway timeout after 5000ms (order_id=88213)

# TRACE (simplified) - the path of ONE request across services
checkout-api (120ms)
  └─ inventory-service (15ms)
  └─ payment-service (98ms)  ← the slow part

New teams often try to solve every monitoring problem with logs, because logs are the most familiar tool - print statements everyone already knows how to write. That works fine at small scale, but it falls apart once there are dozens of services and thousands of requests per second: nobody can "just read the logs" fast enough to catch a slow trend, and grep does not scale the way a purpose-built time-series query does.

Metrics exist specifically to be cheap enough to check continuously - every 15 seconds, forever - and to answer "is this normal?" at a glance from a graph, which is the question a human actually needs answered first, before they go digging into logs or traces for the details.

Tip: If you only remember one rule from this section: reach for a metric first when the question is "how much / how often / is this trending up?", and reach for logs or traces only once a metric has already told you where to look.

Goal: Put this to work in the promgraf-configure-scrape-target lab. Open /labs/prometheus-grafana, pick promgraf-configure-scrape-target, and fix the real broken monitoring stack - a genuine Prometheus server scraping real targets, and a genuine Grafana instance querying it.