Position and Transformer Families

Attention needs order information

Self-attention by itself treats inputs like an unordered set. Positional encoding injects token order, using fixed functions or learned position-related representations. Without it, dog bites person and person bites dog contain the same token set and are difficult to distinguish structurally.

An encoder-only Transformer builds bidirectional representations useful for classification and retrieval. A decoder-only Transformer uses causal masking and generates tokens left to right; most conversational LLMs follow this family. An encoder-decoder Transformer encodes an input sequence and generates a separate output sequence, useful for translation and summarization.

encoder-only:    all input positions can attend to each other
decoder-only:    each position sees only itself and the past
encoder-decoder: encoded source + causal generated target

Scenario: A next-token model is trained without a causal mask. Validation loss looks impossibly good because each position sees future answer tokens. At generation time those future tokens do not exist, and performance collapses - architectural leakage.

Note: Long contexts challenge positional generalization and quadratic attention cost; a declared context window is an operational limit, not unlimited memory.