KV Cache and the CPU-Safe vLLM Simulation

Reuse attention work

During transformer attention, each processed token produces key (K) and value (V) tensors. A KV cache retains those tensors so decoding the next token does not recompute attention state for the entire earlier sequence. Cache memory grows with active sequences and context length.

Prefix caching can reuse state when requests share an identical prefix, such as a long system prompt. A cache hit avoids repeated prefill work. A miss performs the work and may insert the prefix. Least recently used (LRU) eviction removes the entry that has gone unused longest when space is needed.

Analogy: A chef keeps frequently used sauce bases prepared. Reusing the base saves work, but refrigerator space is finite, so the least recently requested container leaves first.

Real vLLM requires a supported accelerator and substantial memory, which this shared two-CPU lab host does not provide. The lab therefore uses a clearly labeled deterministic CLI simulator. It models dynamic batches, prompt compute saved by prefix hits, cache capacity, and LRU eviction. Its numbers are educational metrics, not a hardware benchmark or the vLLM API.

Goal: Tune the simulator from serial, cache-free serving to a bounded batch and LRU prefix cache. Prove cache hits, fewer scheduling batches, and at least twice the baseline simulated throughput.