KV Cache and the CPU-Safe vLLM Simulation

Reuse attention work

Inside the transformer architecture that powers modern LLMs, every token that gets processed produces two internal tensors used by the model's attention mechanism, conventionally called key (K) and value (V). A KV cache is exactly what it sounds like: a stored record of those K and V tensors, kept around specifically so that decoding the next token doesn't require recomputing attention over the entire preceding sequence again from scratch - reusing that earlier work instead of redoing it is precisely what makes fast token-by-token generation practical at all. The catch, echoing the capacity-planning module from earlier in this track: this cache's memory footprint grows directly with both how many sequences are actively being processed at once and how long each of their contexts is.

Prefix caching takes this idea one step further: when two different requests happen to share an identical starting prefix - a long, unchanging system prompt reused across many requests is the classic case - the KV cache computed for that shared prefix in one request can be directly reused for the next one, entirely skipping the redundant prefill work the earlier scenario described. A cache hit means that reuse actually happened, avoiding real recomputation; a cache miss means the work had to be done fresh, and the result may then get inserted into the cache for a future request to potentially reuse. Since cache space is finite, something has to eventually be evicted to make room for new entries - least recently used (LRU) eviction is a common, simple policy that specifically removes whichever cached entry has gone the longest without being reused.

Analogy: A busy kitchen keeps a few frequently used sauce bases pre-made and ready to go. Reusing one saves real, meaningful prep time versus making it from scratch again - but the refrigerator only has so much shelf space, so when something new needs to go in, whichever prepared base has sat untouched the longest is the one that gets cleared out first to make room.

Real vLLM is designed around a supported hardware accelerator (a GPU) and expects a substantial dedicated memory budget - neither of which this platform's shared, two-CPU lab host actually provides, so running the genuine library here simply isn't realistic. Instead, this lab uses a clearly labeled, fully deterministic command-line simulator that models the concepts directly: it simulates dynamic batches forming and draining, the prefill work saved by a prefix-cache hit, a fixed cache capacity, and LRU eviction kicking in once that capacity is exceeded. The numbers it reports are genuinely useful educational metrics for understanding the mechanism - they are not a real hardware benchmark, and the simulator does not implement the actual vLLM API.

Goal: Tune the simulator starting from serial, cache-free request handling all the way up to a properly bounded batch size combined with an LRU prefix cache. Prove, with the simulator's own reported numbers, that cache hits are actually occurring, that fewer overall scheduling batches are needed, and that simulated throughput reaches at least twice the original serial baseline.