Designing Honest Fallbacks

Fallback changes behavior

A fallback sends a request to an alternate deployment after the primary fails. It can improve availability for timeouts, unavailable models, or transient provider errors. It should not retry invalid input, authentication failure, or a request that violates policy.

The fallback model may be smaller, less capable, slower, more expensive, or subject to a different privacy boundary. Keep sensitive local traffic from silently crossing to a hosted provider. Return or log which route served the answer so operators can explain quality changes.

Warning: Unlimited retries and circular fallbacks amplify an outage. Set a small retry budget, deadline, and acyclic route graph.

In the offline lab, the primary alias maps to Qwen2.5 0.5B and the fallback alias maps to SmolLM2 135M. Both are baked into the image and use loopback Ollama, so fallback changes capability but not data residency.

Tip: Use a route alias in client requests. If clients request a provider model directly, the proxy cannot consistently apply the intended routing policy.