Inference, Prompts, and Context

Using a trained model

It helps to clearly separate two very different stages in an LLM's life. Training is the expensive, one-time (or occasional) learning stage where the model's billions of parameters get adjusted, bit by bit, by processing a very large collection of text - it can take enormous computing clusters and weeks of time. Inference is everything that happens after training: the already-trained model reads a prompt and produces an answer, using the parameters it already has, without changing them. Every exercise in this track only ever performs inference - training a modern LLM from scratch is far beyond what a shared two-CPU learning server could do.

Every model has a context window: the maximum number of tokens it can consider at once in a single request. Think of it as the model's desk space - not a filing cabinet with unlimited room, but a fixed-size workbench. System instructions, earlier chat messages, any retrieved reference documents, and the answer being generated all have to fit on that same desk at the same time. When there's too much to fit, something has to give: either the request is rejected outright, or the application must deliberately trim older or less important text to make room.

Analogy: A context window is a desk, not a filing cabinet. You can keep adding papers to the desk, but past a certain point an older paper gets pushed off the edge and is gone from view - the model literally cannot see text that no longer fits.

The prompt is simply the text you send the model to work from. In a real application it's usually assembled from several pieces: a system instruction (the application's own rules, like "answer only from the provided context"), the user's input, and sometimes reference material pulled in from elsewhere. It's good practice to clearly mark where untrusted text (like something a user typed, or a document fetched from a database) begins and ends, so the model - and anyone reading the prompt later while debugging - can tell instructions apart from data. If another program is going to read the model's answer automatically, it's also worth explicitly asking for a specific structured format, such as JavaScript Object Notation (JSON), rather than free-form prose.

Tip: Start with the smallest, plainest prompt that expresses the job clearly. Add extra examples, longer instructions, or stricter formatting constraints only once you've actually tested that they measurably help - a bloated prompt eats into your limited context window for no real benefit.