TTFT Versus Total Latency

Users experience more than one clock

Latency is the general term for how much elapsed time an operation takes - but for an LLM response specifically, a single "how long did it take" number actually hides two quite different user experiences, and it's worth separating them clearly. Time to first token (TTFT) measures the time from sending a request until the very first piece of generated text arrives back. Total latency measures the time until generation is completely finished, start to end. The reason this distinction matters in practice: a streaming interface (recall the streaming concept from the Ollama API module earlier in this track) can start showing a user the first few words almost immediately, even while the rest of a long answer is still being generated behind the scenes - making the perceived wait feel far shorter than the total wait actually is.

Two related, finer-grained measures round this out. Inter-token latency measures the gaps between successively generated tokens once streaming has started - a model that starts fast but then stutters has a good TTFT and poor inter-token latency. Tokens per second is essentially the inverse view of that same idea, describing raw generation throughput. TTFT itself is shaped by several things happening before the first token can appear: how long the request sat queued behind other work, how long the model spent processing ("reading") the input prompt, and the cost of that very first decoding step. Total latency, on top of all that, keeps growing the longer the requested output is.

import time
start = time.perf_counter()
# ... make the request, capture first-token time separately ...
elapsed = time.perf_counter() - start

When actually measuring any of this, use monotonic timestamps like Python's time.perf_counter() rather than reading the regular system clock - a monotonic clock always moves steadily forward and can't be corrupted by something like an NTP time-sync adjustment happening to occur mid-measurement, which could otherwise make an elapsed-time calculation come out negative or wildly wrong. Alongside the raw timing numbers, capture whether the request succeeded, which model answered, the input size, the context-window setting used, and the resulting output length - timing numbers without that surrounding context are hard to compare meaningfully later.

Analogy: At a restaurant, TTFT is the time until the very first dish reaches your table; total latency is the time until the entire meal, every course, has been served. Both genuinely matter to how the meal feels, but they describe two different kinds of frustration if either one is too slow.