Diagnose Slow Starts

Reduce work before blaming the model

When TTFT feels slow, the instinct is often to blame the model itself - but several configuration mistakes, all fixable without touching the model, are common enough to check first. Long prompts increase prefill: the up-front work of processing every input token before generation can even begin. A wrapper that carelessly repeats a lengthy system prompt or reference handbook hundreds of times across many requests wastes real CPU on redundant prefill work every single time. Requesting an oversized context window, larger than the request actually needs, can reserve more runtime memory than necessary and slow things down for no benefit. And running multiple requests in parallel means they're all competing for the same limited CPU capacity on this shared host, which directly raises how long each one waits queued before its own processing even starts.

The concrete fixes mirror those causes closely: use the smallest context window that safely fits the instructions, retrieved evidence, and expected output - not a generously oversized one "just in case." Bound how much text gets stitched into a prompt during construction, stream responses back to the client whenever the interface supports it, cap the maximum number of output tokens a request can request, keep the model loaded in memory rather than reloading it repeatedly when traffic actually justifies the memory cost of doing so, and set the concurrency limit from measured real capacity rather than a guess.

When comparing benchmark results against each other, make sure you're actually comparing like with like: the same model tag, the same prompt, the same temperature setting, the same context-window size, similar host load at the time, and enough runs to be statistically meaningful. Explicitly label "warm" runs (model already loaded) separately from "cold" runs (model just loaded for the first time), since mixing the two together produces a misleading blended number. And report the raw, honest numbers you actually measured on your own machine rather than asserting that one favorable result from a single development laptop constitutes a formal service-level objective for production.

Scenario: A wrapper script, written without much thought for performance, repeats a short prompt 200 times in a benchmarking loop, requests a full 32K-token context window for every single call regardless of actual need, and has streaming disabled entirely. Fixing all three of those settings - trimming the redundant repetition, right-sizing the context window, and enabling streaming - visibly improves the user-perceived wait time without changing which model is being used at all.

Warning: A faster answer is worthless if it's truncated mid-sentence or simply wrong. Always track correctness and latency together as a pair - optimizing purely for speed while ignoring whether the output is still actually correct and complete is a common and costly mistake.

Goal: Repair the benchmark's configuration and retain its structured latency report as real operational evidence - proof, not just a claim, that the fix actually worked.