Continuous Batching

Refill the batch while generation runs

Continuous batching is a scheduler technique that combines token-generation work from active requests and admits new requests as others finish. Unlike static batching, membership can change between decoding steps. Engines such as vLLM use this idea to keep expensive compute resources busy.

A larger maximum batch can improve throughput by sharing scheduling and parallel compute, but consumes more memory and may delay an individual request. Admission control must cap queued work. Production schedulers also consider prompt length, decode state, priorities, and memory blocks rather than simply slicing a list.

Warning: Batch size is not a magic speed dial. If memory pressure causes cache eviction or out-of-memory failure, a larger value reduces reliability and can lower real throughput.

Measure with a workload that resembles production: prompt-length distribution, output lengths, arrival rate, shared prefixes, and streaming clients. Report percentiles such as p95 latency, not only a favorable average.

Tip: Tune one constrained dimension at a time and keep a baseline. A faster result without the same workload and output budget is not a valid comparison.