TTFT Versus Total Latency

Users experience more than one clock

Latency is elapsed time for an operation. Time to first token (TTFT) measures from sending a request until the first generated text arrives. Total latency measures until generation finishes. A streaming interface can display early tokens while the rest are still produced.

Inter-token latency measures gaps between generated tokens, while tokens per second describes generation throughput. TTFT is influenced by request queuing, prompt processing, model loading, and the first decoding step. Total time also grows with output length.

Record monotonic timestamps such as Python's time.perf_counter() so system-clock adjustments do not corrupt elapsed measurements. Capture success, model, input size, context setting, and output length beside timing.

Analogy: At a restaurant, TTFT is time until the first dish arrives; total latency is time until the entire meal is served. Both matter, but they describe different frustration.