Decompose the Latency Budget

Find which stage owns the wait

End-to-end latency includes queueing, retrieval, prompt construction, model prefill over input tokens, first-token decoding, remaining output generation, validation, and network transfer. Time to first token (TTFT) captures the wait until generation starts; inter-token latency captures stream smoothness; total latency captures completion.

total = queue 40 + retrieve 35 + prefill 180 + decode 900 + validate 15 = 1170 ms
TTFT  ≈ queue + retrieve + prefill + first decode

Instrument each stage before optimizing. A faster model cannot repair a slow serial retrieval query, while aggressive output limits do nothing for excessive prompt prefill. Track percentiles by route and input size because averages hide overloaded tails.

Analogy: A trip includes waiting for a taxi, driving, security, and boarding. Buying a faster plane may barely change door-to-door time when the queue is the bottleneck.

Tip: Set a latency budget per stage and an overall deadline. Cancellation should propagate so work stops after the caller no longer needs it.