Timeouts, Limits, and Observability

Bound every expensive dimension

A local inference API on a shared, CPU-only host needs firm limits on every dimension that could turn into unbounded work: maximum prompt length, maximum output tokens, how many requests can run concurrently, and a maximum request duration. When one of those limits is hit, the right response is to reject or cancel the excess work outright - not to quietly build an ever-growing internal queue with no ceiling, which just delays the same overload problem instead of solving it. HTTP status codes in the 4xx range specifically mean the client sent something invalid or unauthorized; a client that gets a 4xx and simply retries the exact same request will keep getting the exact same failure, because retrying doesn't fix a malformed request.

For visibility into what's actually happening, log a request ID (so one specific request's full journey can be traced through the system), the model tag used, the request's duration, token counts when available, and its final result status. Just as important is what not to log: never write full prompts or authorization headers into logs, since prompts can easily contain sensitive user data. Finally, measure both the average response time and the tail latency - specifically the slowest fraction of requests, like the 95th or 99th percentile - because a comfortable-looking average can completely hide a meaningful chunk of users having a genuinely painful, multi-second wait.

Scenario: An endpoint with no input-size limit accepts a 20 MB prompt from a client, and processing that enormous prompt ties up the host's only available CPU for several minutes, effectively taking the service down for everyone else. Adding a simple maximum-prompt-length check turns that slow, confusing outage into an immediate, clear 422 ("unprocessable input") response the client gets back in milliseconds.