Retries, Fallbacks, and Cost

Handle failures by type

Not every failure deserves the same response. Only retry genuinely transient failures - problems that are likely to disappear on their own with a fresh attempt, like a brief network hiccup or a momentary connection reset. Exponential backoff means waiting progressively longer between each retry attempt rather than hammering the same failing endpoint at a fixed interval; jitter adds a small random amount of extra delay on top of that, specifically so that many clients retrying at once don't all pile back in at exactly the same instant and recreate the very overload that caused the failure. Bad credentials, genuinely invalid input, and a prompt that simply exceeds the model's context window should never be retried - the exact same request will fail the exact same way every time, no matter how many attempts you make. Keeping a small, fixed retry budget also matters: without one, a real outage can multiply into far more total traffic than the original request volume, as every failed call spawns retries of its own.

A fallback, distinct from a retry, sends the request to a genuinely different provider or model once the primary one has actually failed. It's a real trade-off, not a pure improvement: the fallback path changes cost, privacy characteristics, response speed, and even the style of the answer you get back. Always record which provider actually answered a given request, and specifically test the fallback path in isolation - don't just assume adding one automatically creates reliability, since an untested fallback path can itself be broken (see the fault-injection module later in this track for exactly this kind of test).

Tip: In these offline labs, only the local Ollama provider is ever actually reachable - there's no live internet access to fall back to. The same code structure and configuration pattern still teaches real multi-provider routing concepts, just without needing network access or a real paid API key to practice with.