Timeouts, Limits, and Observability

Bound every expensive dimension

Limit prompt length, output tokens, requests running at once, and request duration. Cancel or reject excess work instead of building a queue with no maximum. HTTP 4xx status codes mean the client sent an invalid or unauthorized request; retrying the same request will not repair it.

Log a request ID, model tag, duration, token counts when available, and result status. Do not log full prompts or authorization headers. Measure failures and tail latency - the slowest group of requests - as well as the average, because a good average can hide painful delays.

Scenario: A harmless-looking endpoint accepts a 20 MB prompt and ties up the only CPU for minutes. Input validation turns that outage into an immediate, useful 422 response.