Self-Attention, Heads, and Masks

Several relationship maps at once

In self-attention, queries, keys, and values all come from the same sequence, so every token builds a context-aware representation from other tokens. Multi-head attention runs several learned Q/K/V projections in parallel, allowing different heads to capture different relationship patterns before their outputs are combined.

A causal mask prevents a language-model token from attending to future tokens during next-token prediction. Without it, training could peek at the answer. A padding mask prevents artificial blank positions in a batch from contributing attention.

allowed attention in a causal decoder
token 1: [1 0 0 0]
token 2: [1 1 0 0]
token 3: [1 1 1 0]
token 4: [1 1 1 1]

Standard self-attention compares every position with every other, so compute and attention-score memory grow roughly with sequence length squared. This makes long context expensive and motivates efficient-attention research.

Note: This mechanism is the direct conceptual bridge to the existing ai-llm-foundations module, where a trained language model uses attention-based layers to predict tokens during inference.

Warning: Masking mistakes are a form of leakage. If a training position can attend to its future target, loss looks unrealistically good and generation fails when that unavailable future disappears.