LLM Serving Observability: TTFT, Token Accounting, Batching, and Fallbacks

Trace the Whole Request, Not Only the Model Call

An LLM request can spend time in authentication, a queue, prompt construction, retrieval, model loading, first-token generation, streaming, guardrails, and persistence. A single duration metric cannot say which stage regressed.

Create one request ID at ingress and propagate it through retrieval, model proxy, provider or local server, guardrails, and response streaming. Record spans for each stage, but redact prompt and output bodies by default.

Metrics That Explain User Experience

• Time to first token (TTFT): request accepted to first streamed token.
• Inter-token latency / throughput: generation speed after the first token.
• End-to-end latency: includes retrieval, guardrails, and serialization.
• Queue time: exposes saturation separately from slow generation.
• Prompt and completion tokens: capacity and cost drivers.
• Error, timeout, and fallback rate: reliability and routing behavior.

Break these down by model, route, status, and bounded workload class. Never put user IDs or raw prompts in metric labels.

---

Batching and KV Cache Change the Capacity Model

Traditional request batching waits to form a batch, which can hurt TTFT. Continuous batching lets newly arrived work join as active sequences finish. The KV cache avoids recomputing attention state for prior tokens but consumes memory proportional to active sequence length and concurrency.

Watch queue depth, active sequences, cache utilization, rejected requests, and token throughput together. Higher request throughput is not a win if tail TTFT violates the user-facing objective.

Fallbacks Must Be Visible and Compatible

A LiteLLM-style proxy can normalize calls and route across backends. A fallback should activate only for defined transient failures, have a bounded attempt budget, and emit an explicit routing span. Verify that the fallback supports the required context length, tool schema, response format, and data-handling policy.

Do not silently replace a stronger model with a cheaper one and label the request successful. Return the actual served model in internal telemetry and, where product behavior requires it, in response metadata.

Health and Readiness

Liveness means the process responds. Readiness means the configured model, tokenizer, storage, and required dependencies can serve within policy. On startup, remain unready until those checks pass. On repeated fatal inference errors, shed traffic, preserve diagnostics, restart cleanly, and prove recovery with a real bounded inference probe.

The labs Trace Every LLM Request, Simulate Continuous Batching, Repair Fallback Routing, and Recover Service Health exercise the complete path.