Why LLM Serving Is Different

Generation is a changing workload

A large language model (LLM) first processes all input tokens in a prefill phase, then produces output one token at a time in a decode phase. Requests have different prompt and answer lengths, so they do not finish together. A fixed batch waits for its slowest member and leaves capacity unused as shorter requests complete.

Throughput is useful work completed per unit of time, often tokens or requests per second. Latency is how long one request waits. Time to first token measures responsiveness; inter-token latency measures streaming smoothness. Maximizing throughput can increase queueing latency, so an operator chooses targets rather than one universal maximum.

Analogy: A fixed school bus leaves only when every seat is filled and returns only after every passenger finishes. A continuous shuttle admits new passengers whenever a seat opens.

Scenario: Six requests run serially on an accelerator. Each pays scheduler overhead, repeated system prompts are recomputed, and users wait even though memory could hold several active sequences.

Capacity planning includes model weights, temporary compute buffers, the key-value cache, request queues, and the serving process. A configuration that fits at low traffic can run out of memory under longer contexts or greater concurrency.