Inside a Transformer Block

Attention routes; feed-forward layers transform

A Transformer stacks blocks containing self-attention and a position-wise feed-forward network, a small multi-layer network applied independently to each position. Attention mixes information across positions; the feed-forward part transforms each position's resulting features.

Each sublayer usually sits inside a residual connection, adding its input back to its output, and layer normalization, rescaling internal activations for stable training. Residual paths let information and gradients travel through deep stacks without depending entirely on every transformation.

token states
  -> layer norm -> multi-head attention -> add residual
  -> layer norm -> feed-forward network -> add residual
  -> next Transformer block

Analogy: Attention is a conference where positions exchange relevant notes. The feed-forward network is each participant privately processing the notes. Residual connections preserve the original notebook beside every revision.

Tip: Do not reduce a Transformer to attention alone. Embeddings, positional information, feed-forward layers, normalization, residuals, and output projection are all necessary parts of the system.