Repeatable Inference Operations

Control the variables

If a teammate can't reproduce a result you got from a model, the most common reason is that some detail wasn't written down. A reproducible inference call records: the exact model tag (its name and version, since qwen2.5:0.5b today might not be byte-identical to qwen2.5:0.5b after an update), the reusable prompt template used to build the request, whatever context was supplied, the generation settings, and the version of your own application code that made the call.

One especially important generation setting is temperature: a number that controls how "adventurous" the model is when choosing its next token. At low temperature, the model almost always picks the single most likely next token, which makes output far more repeatable - ideal for extraction tasks and automated tests where you want the same input to reliably produce the same output. At higher temperature, the model is more willing to pick a less-likely-but-still-plausible token, which produces more varied, creative-feeling text at the cost of consistency.

Beyond that, a few operational habits matter a lot on a CPU-only host: set a timeout so a single stuck model call can never hang forever; cap both input and output length so one oversized request can't monopolize the machine; and keep concurrency - how many requests run at the same time - low, since a CPU-only host has far less parallel headroom than one with a GPU.

Finally, distinguish two related but different checks. A health check only proves the Ollama daemon is up and answering some request at all. A stricter readiness check additionally proves that the specific model your application actually needs is installed and loadable - a distinction that matters a great deal, as the next scenario shows.

Scenario: A deployed API's process-health check passes with a green checkmark, yet every single real request to it fails. The cause: the health check only pinged the Ollama daemon itself, never verifying that the configured model tag was actually installed. A proper readiness check - one that also confirms the required model exists - would have caught this inventory mismatch immediately, before any real traffic arrived.

Goal: Repair the wrong-model incident in the graded AI lab by correcting the configuration and proving the tiny, approved model actually serves requests - not just that the daemon is technically alive.