Prove Routing with Fault Injection

A configured fallback is not yet a tested fallback

An ordinary smoke test - simply sending one request through and confirming you get a reasonable-looking answer back - only ever proves that the primary path works. It tells you nothing at all about whether the fallback path, sitting unused most of the time, would actually work correctly if it were ever really needed. Fault injection is the practice of deliberately making a dependency fail under fully controlled, intentional conditions, and then checking that the service genuinely follows its documented recovery policy in response - common ways to do this include pointing at a model tag you know is unavailable, closing off a port the service expects to reach, forcing a timeout, or substituting in an explicit test stub in place of the real backend.

$ cp litellm_config.yaml /tmp/test_config.yaml
# sabotage only the temporary copy's primary model
$ sed -i 's/qwen2.5:0.5b/qwen2.5:missing-tag/' /tmp/test_config.yaml
$ litellm --config /tmp/test_config.yaml --port 4001 &
$ curl -s http://127.0.0.1:4001/chat/completions -d '{"model":"local-primary", ...}'

The right way to run this test preserves your actual, correct source configuration untouched throughout: create a temporary copy of it, sabotage only the primary backend entry in that copy (as shown above), start the proxy pointed at that broken temporary copy on a separate test port, and then send a completely real chat-completion request through it. If everything is wired correctly, that request can now only possibly succeed by actually going through the configured fallback route - there's no other way for it to get a real answer.

Scenario: An operator reviews the proxy's YAML configuration, sees both the primary and fallback model names listed there, and confidently declares resilience "done." During the very first real outage, requests still fail anyway - because the fallback mapping, on closer inspection, had actually been attached to the wrong alias entirely, a mistake that a config file review alone, without ever actually testing it under a real simulated failure, completely failed to catch.

When checking the result, validate more than just the HTTP status code coming back - confirm the actual response shape is correct too, since a real chat completion needs to contain a genuinely non-empty assistant message, not just a technically-200 response with an empty body. It's also worth inspecting the proxy's own logs at this point, specifically to confirm they show both the initial primary failure being detected and the subsequent fallback selection actually happening - and remember to cleanly stop the temporary test proxy process once the test is done, regardless of whether it passed or failed.

Goal: Repair the LiteLLM model list and its fallback mapping. The lab's checker starts the real, actually-installed proxy, sabotages only a temporary copy of the primary configuration, and proves that SmolLM still successfully returns a valid completion through the fallback path.