Measure a Real Ollama Stream

Observe the first response event

Ollama's local /api/generate endpoint, introduced back in the Ollama CLI module, can return its response as newline-delimited streaming JSON rather than one single completed document - exactly the streaming mode covered there. To actually benchmark TTFT for real, start a timer before making the HTTP request, then read the response body incrementally, line by line, as it arrives. Capture the TTFT the moment the first line containing actual generated text shows up, and stop a separate total-time timer only once a line marked "done": true arrives, signaling the model has finished.

$ curl -s -N http://127.0.0.1:11434/api/generate \
  -d '{"model":"qwen2.5:0.5b","prompt":"Explain TCP in one sentence","stream":true}'
{"response":"TCP","done":false}
{"response":" is","done":false}
{"response":" a","done":false}
...
{"response":"","done":true,"total_duration":812345678}

It's important to run this benchmark more than once before drawing any conclusion, because the very first request in a session often pays an extra cost for actually loading the model into memory - a one-time cost that has nothing to do with the model's steady-state speed. Report both individual run times and the median (the middle value once results are sorted) rather than the average, since the median is far less distorted by one unusually slow outlier run. Resist the urge to invent one universal "pass" threshold in milliseconds that's supposed to apply across every different CPU this might run on - a number that's fast on one machine can be genuinely slow on another with weaker hardware.

A non-streaming request ("stream":false) only returns after the entire answer is already complete, which means it structurally cannot demonstrate a real interactive TTFT at all - there's no meaningful "first token arrived" moment to measure separately from "everything arrived." A subtler trap: even a model that streams correctly internally can have its streaming effectively defeated by an application-level proxy or wrapper in front of it that buffers the whole response before passing anything along - from the client's perspective, that still looks and feels exactly like a non-streaming response.

Tip: Always benchmark through the same wrapper and network path your real users' requests actually travel through. A benchmark that talks directly to the model, versus one that goes through your full application stack, are answering two genuinely different questions - and only the second one tells you what users actually experience.

Goal: Run two real requests against the pre-baked Qwen2.5 0.5B model and verify each request's measured first-token time is meaningfully below its own total time - direct, hands-on proof that streaming is actually working as intended.