Inference, Prompts, and Context

Using a trained model

Training is the expensive learning stage that adjusts the model's parameters from a very large set of examples. Inference is the later usage stage: the trained model reads a prompt and produces an answer. These labs only perform inference. Training a modern LLM needs far more computing power than this two-CPU learning server.

The context window is the maximum number of tokens the model can consider in one request. Think of it as the model's workbench. Instructions, earlier chat messages, retrieved documents, and the new answer all compete for the same space. When they do not fit, the client must reject the request or deliberately remove some text.

A prompt is the text sent to the model. It often includes a system instruction (the application's rules), user input, and reference material. Mark where untrusted text begins and ends. If another program will read the result, ask for a specific format such as JavaScript Object Notation (JSON).

Analogy: A context window is a desk, not a filing cabinet. Adding more papers eventually pushes an older paper off the desk.

Tip: Start with the smallest prompt that expresses the job. Add examples or constraints only when tests show that they help.