Tokens and Next-Token Prediction

A language model is a prediction engine

Analogy: Phone autocomplete guesses the next word from what you already typed, based on patterns it has seen before. A large language model, or LLM, does the exact same basic act - guess the next piece of text - but with a vastly richer statistical model built from an enormous amount of text.

An LLM doesn't read text as whole words the way you do. It first breaks the text into tokens: small chunks that might be a whole common word, a piece of punctuation, or a fragment of a longer or rarer word. "unhelpful", for example, might become three tokens: un, help, ful. Generation then works one token at a time, in a loop: look at every token so far, calculate a probability for each possible next token, pick one, tack it onto the text, and repeat.

prompt:  "The capital of France is"
step 1:  model scores every possible next token
         " Paris"  -> 0.86
         " a"      -> 0.04
         " located"-> 0.03    ...
step 2:  picks " Paris", appends it, and repeats from the new, longer text

It's worth sitting with that mechanism, because it explains a lot of LLM behavior that otherwise looks mysterious: the model is not looking anything up in a fact database unless your application explicitly hands it one (that's what retrieval-augmented generation, covered later in this track, is for). It is predicting plausible continuations of text, based on patterns learned from its training data.

Underneath, an LLM is a neural network: a computer system built from many layers of small connected calculations. Each of the countless numbers the network learned during training is called a parameter. Qwen2.5:0.5b, one of the two tiny models pre-installed on this platform, has about 0.5 billion parameters; SmolLM2-135M has about 135 million. More parameters generally let a model capture richer patterns, but they also demand more memory and CPU time to run - which is exactly why this offline, CPU-only platform deliberately sticks to small models instead of trying to squeeze in something far larger.

Scenario: You ask a model when a particular server was last restarted. It confidently writes out a specific date and time - but it has no access to that server's logs. The answer sounds plausible because the model is good at producing plausible-sounding text, and it is still completely invented.

Warning: A confident but unsupported answer like that is called a hallucination. It isn't a rare bug; it's a natural consequence of how next-token prediction works when the model has no real source of truth to draw from. Real systems that need factual answers connect the model to trusted data (see the RAG modules later) and validate important output before acting on it.