Logs, Metrics, and Traces

Observe what the application actually did

Everything built so far in this track - the RAG pipeline, the FastAPI service, the terminal agent - is a real, running system now, and running systems eventually misbehave in ways nobody predicted in advance. Observability is the general ability to understand what a system actually did internally just by looking at what it produces on the outside, without having to guess or reproduce the problem live. Three signals show up again and again in observability work, and it's worth being precise about how they differ. A log records one discrete event, usually as a line of text or structured data, at the moment it happened. A metric is a number, aggregated over time - a request count per minute, an average response time. A trace follows one single request as it moves through multiple steps of a system, connecting everything that happened because of that one request.

An LLM request in a real application is rarely just "call the model and get an answer" - it typically passes through several distinct steps: authentication, retrieval (pulling relevant context, as covered in the RAG modules), prompt construction, the actual model inference call, and output validation. To make sense of all that later, give the whole request a single unique trace ID at the moment it starts, and pass that same ID through every step. Without it, you're left with a pile of separate timestamped log lines from different parts of the system with no way to tell which ones belong to the same request, especially once many users are making requests at the same time.

{"trace_id": "a1f9-...", "ts": "2026-08-30T10:14:02Z", "step": "retrieval", "latency_ms": 42}
{"trace_id": "a1f9-...", "ts": "2026-08-30T10:14:02Z", "step": "inference", "model": "qwen2.5:0.5b", "input_tokens": 310, "output_tokens": 48, "latency_ms": 890}

Useful fields to record on each step include a timestamp, the trace ID, the model version used, input and output token counts, an estimated cost, the step's latency, its status (success or failure), any retrieval IDs involved, and which version of the prompt template was in play. Store this as structured JSON, like the two lines above, rather than free-form prose sentences - prose is comfortable to write but painfully fragile to parse back out reliably later, exactly when you need it most.

Analogy: A parcel's tracking number lets a delivery company connect a whole scattered pile of warehouse scan events - scanned in Memphis, scanned in Chicago, out for delivery - into one coherent journey. Without that shared number, the exact same scan events would just be an unconnected pile of timestamps that can't explain where your particular parcel actually waited.

Warning: Observability is not a license to copy anything and everything into a log just because it might be useful someday. Never dump sensitive prompts, personal user data, API keys, or authorization headers straight into trace records. Prefer metadata, hashes of sensitive values instead of the values themselves, deliberate redaction (covered in depth in the security phase later in this track), tight access control on who can read the logs, and short retention windows.