Running Local LLMs with Ollama: A Practical Operations Guide

Local Inference Is Still a Service

Ollama makes the first prompt easy, but production habits begin after the demo works. A reliable local inference workflow identifies the exact model, verifies that the server is ready, bounds requests, records latency, and separates machine-readable output from diagnostics.

Start with inventory and health rather than assuming a model name from a laptop still exists on the target host:

ollama list
curl --fail --silent http://127.0.0.1:11434/api/tags | jq '.models[].name'

An empty model list is different from an unreachable server. Preserve that distinction in monitoring and incident notes.

---

Make Requests Repeatable

Use the HTTP API when another program consumes the response. Pin the model name and make streaming an explicit choice:

jq -n --arg model 'tinyllama:latest' --arg prompt 'Return one JSON object with a summary field.'   '{model:$model,prompt:$prompt,stream:false}' |
curl --fail --show-error --silent   --connect-timeout 2 --max-time 60   -H 'Content-Type: application/json'   --data-binary @- http://127.0.0.1:11434/api/generate |
jq -e '.response'

Do not build JSON by interpolating untrusted prompt text into a quoted shell string. Let jq encode it. Keep diagnostics on stderr so stdout remains safe for pipelines.

Measure the Right Latency

Total latency hides two different user experiences. Time to first token reflects queueing, model loading, prompt processing, and cold start. Generation time reflects token throughput after output begins. Record both, together with prompt and completion token counts, model identity, and whether the request was warm.

For capacity planning, measure resident memory and concurrent-request behavior. Quantization reduces memory and can make CPU inference practical, but it can also change output quality. Treat a GGUF file and quantization level as versioned model identity, not an invisible optimization.

Operate Failure States Deliberately

• A missing model should fail clearly, not silently route to an unknown alternative.
• A timeout should identify whether the request queued, loaded, or generated slowly.
• A health endpoint should prove the process is alive; readiness should prove the configured model can serve.
• Retries should be bounded and used only for transient failures.
• Prompts and outputs may contain secrets or personal data, so tracing needs redaction and access control.

Practice the failure modes in AI Engineering Labs, especially Recover the Wrong Ollama Model, Benchmark Time to First Token, and Restore Service Health.