Route and Cache Deliberately

Spend capability where it changes outcomes

A model router selects a model based on task, risk, latency, privacy, or budget. Simple classification may go to a small local model; difficult synthesis may use a stronger approved model. Routing must be evaluated end to end because a weak router can erase savings or send sensitive data across the wrong boundary.

Caching can reuse deterministic or slow-changing results. A cache key must include model version, prompt-template version, normalized input, relevant generation settings, and retrieved-data version. Otherwise stale or semantically different requests collide. Sensitive outputs need access-controlled caches and retention rules.

cache key = hash(model + prompt_version + input + settings + corpus_revision)

Warning: Do not cache non-idempotent tool actions or personalized answers under a shared key. A technically correct hit can become a privacy leak.

Note: The existing latency/TTFT and continuous-batching modules deepen measurements and scheduler behavior after this architecture-level view.