Vanishing and Exploding Gradients

Repeated multiplication can erase or amplify signal

Backpropagation multiplies local derivatives through many layers. Repeated values below 1 can shrink a gradient toward zero, producing vanishing gradients that leave early layers learning extremely slowly. Repeated large values can create exploding gradients, causing unstable updates and numeric overflow.

0.5 multiplied through 10 layers ≈ 0.001
2.0 multiplied through 10 layers = 1024

ReLU-family activations, careful initialization, normalization, residual connections that provide short additive paths, and architectures such as LSTMs help gradients travel. Gradient clipping caps an overly large gradient norm before an update, treating the symptom of explosion while the team investigates its cause.

Scenario: Loss suddenly becomes NaN, meaning not a valid number. Logs show gradient norms jumping from 3 to 100,000. Clipping and a lower learning rate stabilize training, while data checks rule out invalid inputs.

Warning: A stable training loss does not prove useful gradients reach every layer. Inspect gradient statistics when deep components appear frozen.