Designing Honest Fallbacks

Fallback changes behavior

A fallback, in this proxy context, sends a request on to an alternate deployment specifically after the primary one has genuinely failed. Used well, it can meaningfully improve availability against timeouts, an unavailable model, or transient provider errors - the same category of genuinely-worth-retrying failure the LiteLLM module earlier in this track already distinguished from failures that should never be retried. Consistent with that earlier distinction, a fallback should never trigger for invalid input or an authentication failure, or for a request that simply violates policy outright - none of those get fixed by trying a different backend.

It's worth being upfront that a fallback model is very often a genuinely different thing from the primary, not just a spare copy of it: it may be smaller and less capable, noticeably slower or faster, priced differently, or even subject to an entirely different data-privacy boundary if it happens to be hosted somewhere else. Keep sensitive traffic that's meant to stay strictly local from silently crossing over to a hosted fallback provider without an explicit, deliberate decision to allow that. And always record, per request, exactly which route actually served the final answer - so an operator investigating a quality complaint later can immediately tell whether the primary or the fallback path was responsible.

Warning: Unlimited retries combined with fallback routes that loop back on each other can actively amplify an outage rather than mitigate it - a burst of failing requests can multiply into far more total load than the original traffic ever was. Set a small, fixed retry budget, an overall deadline for the whole attempt, and make sure the graph of fallback routes is strictly acyclic (no route can eventually fall back to itself, even indirectly).

In this platform's offline lab environment specifically, the primary alias maps to Qwen2.5 0.5B and the fallback alias maps to SmolLM2 135M - the same two models introduced all the way back in the very first module of this track. Both are already baked directly into the lab image and both are reached through the same local, loopback Ollama daemon, so in this particular setup a fallback changes which model's capability answers the request, but never changes where the data itself actually goes.

Tip: Always have client requests target a route alias like local-primary, never a specific provider model name directly. If clients bypass the alias and request a concrete provider model themselves, the proxy has no way left to consistently apply its own intended routing and fallback policy on their behalf.