Why LLM Serving Is Different

Generation is a changing workload

Everything in this track so far has mostly considered one request at a time. This final phase asks a harder, more realistic question: what happens when a real serving system has to handle many requests at once, arriving unpredictably, with wildly different lengths? A large language model (LLM) processes a request in two distinct phases. First, in the prefill phase, it reads and processes every input token from the prompt all at once. Then, in the decode phase, it produces output one token at a time, in the loop described all the way back in the first module of this track. Different requests naturally have different prompt lengths and different answer lengths, so a batch of requests started together will not, in general, finish together. If a serving system naively waits for a whole fixed batch to fully finish before starting any new work, the requests that finished early sit idle, wasting capacity that could otherwise already be serving someone else.

Throughput is how much useful work a system completes per unit of time - often measured in tokens or full requests per second. Latency, familiar from the observability phase earlier in this track, is how long any single request has to wait. Time to first token (from the latency-and-TTFT module) measures how responsive the system feels; inter-token latency measures how smoothly a stream continues once it's started. Here's a genuine trade-off worth internalizing: pushing for maximum throughput - keeping every bit of compute capacity constantly busy - can increase how long individual requests sit queued waiting their turn, so a real operator has to deliberately choose a target balance between the two rather than chasing one number in isolation.

Analogy: A fixed-schedule school bus leaves only once every seat is full, and doesn't return to the depot until every single passenger has reached their stop - a design that wastes capacity the moment any passenger's stop is closer than the last one's. A continuous shuttle, by contrast, lets a new passenger board the instant a seat becomes free, keeping the vehicle efficiently full the whole time.

Scenario: Six separate requests run one after another, serially, on an expensive accelerator. Each one separately pays the scheduler's fixed per-request overhead, a shared system prompt gets fully recomputed from scratch for every single request instead of being reused, and users end up waiting even though the hardware's memory could easily have held several requests' worth of active state at the same time.

Capacity planning for a real serving system has to account for several things stacked together: the model's own weights, temporary compute buffers, the key-value cache (covered in depth two sections from now), request queues waiting their turn, and the serving process's own overhead. A configuration that comfortably fits in memory at low traffic can genuinely run out of memory once contexts get longer or concurrency increases - the same underlying capacity-planning lesson from the GGUF-and-quantization module earlier in this track, just now applied at the scale of many simultaneous requests instead of one.