Queries, Keys, and Values

Ask, match, retrieve

Each token representation is projected into three vectors. A query (Q) describes what the current position is looking for. A key (K) describes what each candidate position offers for matching. A value (V) carries the information to retrieve if that candidate receives weight.

Consider one query q=[1,0], two keys k1=[1,0], k2=[0,1], and values v1=[10,0], v2=[0,6]. Dot products give scores 1 and 0. Softmax turns them into approximate weights 0.73 and 0.27. The attention output is 0.73×v1 + 0.27×v2 = [7.3,1.62].

scores  = [q·k1, q·k2] = [1, 0]
weights = softmax(scores) ≈ [0.73, 0.27]
context = 0.73 v1 + 0.27 v2 = [7.3, 1.62]

Real scaled dot-product attention divides scores by the square root of key dimension before softmax, preventing large vector dimensions from producing extremely peaked, hard-to-train weights.

Analogy: A query is a search request, keys are catalog descriptions, and values are the actual books. Match against the catalog, then retrieve a weighted blend of content.

Note: Queries, keys, and values are learned projections of token states, not three different copies of the original words with fixed linguistic meanings.