Tune Continuous Batching and KV-Cache

Fix a serving configuration causing terrible simulated throughput.

A advanced AI Engineering challenge worth 20 points. Solve it hands-on in a real AI Engineering environment in your browser - no local setup, no fake shells.

The Challenge

Scenario: A model-serving simulator is running every request one at a time with its KV-cache disabled - the serving equivalent of a restaurant taking one order at a time with no memory of repeat customers.

Configure bounded continuous batching and a real KV-cache with an LRU eviction policy, then prove improved cache-hit rate and throughput against the simulator's own metrics. Run check.