Diagnose Slow Starts

Reduce work before blaming the model

Long prompts increase prefill, the work of processing input before generation. Repeating a handbook hundreds of times wastes CPU. An oversized context window can reserve more runtime memory than the request needs. Multiple parallel requests compete for limited CPU and can raise queue time.

Use the smallest context that safely fits instructions, evidence, and output. Bound prompt construction, stream responses, cap output tokens, keep one model loaded when traffic justifies the memory, and set concurrency from measured capacity.

Compare like with like: same model tag, prompt, temperature, context, host load, and number of runs. Warm and cold starts should be labeled separately. Report raw observations rather than claiming that one development-machine result is a service-level objective.

Scenario: A wrapper repeats a short prompt 200 times, asks for a 32K context, and disables streaming. Fixing all three settings improves the user-visible wait without changing models.

Warning: A faster answer is not useful if it is truncated or wrong. Track correctness and latency together.

Goal: Repair the benchmark configuration and retain its structured latency report as operational evidence.