Configuration and Secrets

Separate routing from code

A common and costly mistake is hard-coding which specific model or provider to use directly into application source code, which then requires a code change and redeploy every time you want to switch. Instead, read model names, base URLs (the starting web address a provider's API is reached at), timeouts, and any API keys from environment variables or a properly protected configuration file, and validate that they're all present and sensible when the application starts up - printing out only safe, non-sensitive details like which provider and model were selected, never the key itself.

Local Ollama, since it only listens on 127.0.0.1 inside the lab and never leaves the machine, needs no secret at all. A provider reached over the public internet, by contrast, almost always requires an API key to authenticate and bill your usage. That key must never end up in shell history, command-line process arguments (visible to other processes on the same machine), source control, an MLflow logged parameter, or a debug log line - all of the places this track has already flagged as places sensitive data quietly leaks.

Scenario: A developer enables an automatic fallback to a hosted provider purely for convenience, so the application "never goes down" even if the local model has a problem. During a routine local outage, private prompts that were meant to stay entirely on-machine start silently leaving to an external service instead, with nobody noticing until much later. Making fallback routing an explicit, deliberate policy decision - rather than a convenient default - prevents exactly this kind of surprise.

Warning: A unified client makes it dangerously easy to route sensitive data to a hosted fallback almost by accident, precisely because switching providers is now so simple. Make any fallback policy explicit in configuration, and keep it disabled by default for any workflow handling sensitive data.