Continuous Batching

Refill the batch while generation runs

Continuous batching is a scheduling technique built to solve exactly the wasted-capacity problem from the previous section: it combines the token-generation work of several currently-active requests together, and actively admits brand-new requests into that same batch the moment other requests finish and free up room - directly unlike static batching, where a fixed group of requests is locked in together from the start and nothing new can join until the whole group is done. Serving engines such as vLLM (the tool this section's simulator is modeled after) rely on exactly this idea to keep expensive compute resources continuously busy rather than idling between fixed batches.

A larger maximum batch size can genuinely improve overall throughput, since more requests get to share the fixed overhead of a scheduling round and any parallel compute available - but it also consumes more memory, and can make an individual unlucky request wait longer buried inside a larger batch. Some kind of admission control has to cap how much work gets queued up at once; real production schedulers also weigh prompt length, how far along each request's decode process already is, request priorities, and available memory blocks, rather than simply slicing a plain list of requests into fixed-size chunks.

Warning: Batch size is not a simple dial you can just turn up for more speed with no downside. Push it too far and memory pressure can force the system to evict cached state early (see the next section) or trigger an outright out-of-memory failure - at which point a larger batch size has made the system less reliable, and can easily result in lower real throughput than a smaller, well-tuned batch would have delivered.

Measuring the effect of any batching change properly means testing with a workload that actually resembles production traffic: a realistic distribution of prompt lengths, a realistic distribution of output lengths, a realistic arrival rate, requests that genuinely share common prefixes (relevant to the next section), and real streaming clients rather than only clients that wait for a full response. Report percentile latencies such as p95 (the value 95% of requests fall under) alongside the average - a comfortable-looking mean can, exactly as in the observability phase, comfortably hide a meaningfully painful tail of slow requests.

Tip: When tuning any one of these settings, change one constrained dimension at a time and keep a fixed, unchanged baseline to compare against. A faster-looking result measured under a different workload or a different output-length budget than your baseline is simply not a valid, trustworthy comparison, however tempting the improved number looks.