Momentum Smooths the Route

Accumulate useful direction

Momentum maintains a moving update velocity that combines recent gradients. Consistent directions accumulate speed, while alternating directions partly cancel. This can accelerate travel along long valleys and reduce side-to-side oscillation.

velocity = momentum_rate × old_velocity + current_gradient
weights  = weights - learning_rate × velocity

If successive gradients point downhill along one axis but alternate across a steep narrow axis, ordinary SGD zigzags. Momentum averages the history into a steadier diagonal route. Too much momentum or an excessive learning rate can overshoot.

Analogy: A rolling ball remembers motion. Small bumps do not reverse it instantly, while a sustained slope builds speed.

Scenario: SGD loss falls slowly with visible oscillation. Adding moderate momentum reaches the same validation loss in fewer epochs, without changing model capacity or data.

Warning: Faster training loss reduction is not necessarily better generalization. Continue monitoring validation evidence and stop appropriately.