Let Each Token Look Where It Needs

Replace one compressed history with direct access

RNNs squeeze prior context through one evolving state. Attention instead lets a position directly calculate how much to use information from other positions. The model creates a weighted mixture of their representations, focusing differently for each token and each layer.

sentence: The server could not start because it was misconfigured.
question at token 'it': which earlier representation matters?
attention may weight 'server' strongly and unrelated tokens weakly

Attention does not literally produce human awareness or a guaranteed explanation. It is a differentiable routing mechanism: compute relevance scores, normalize them into weights, and take a weighted sum. Those weights are learned indirectly because useful routing reduces training loss.

Analogy: A meeting participant has every colleague's notes available. For each new question, they assign different relevance weights to the notes and combine the most useful evidence instead of relying on one fixed summary written hours ago.

Warning: A large attention weight shows strong internal influence in that computation, not necessarily a faithful human-readable reason for the final prediction.