Gradient Descent Step by Step

Adjust parameters to reduce loss

A loss function converts prediction error into one number to minimize. Gradient descent repeatedly calculates the gradient of loss with respect to parameters and subtracts a small multiple of it. The multiplier is the learning rate, controlling step size.

start weight w = 4.0
gradient at w  = +3.0
learning rate  = 0.1
new w = 4.0 - 0.1×3.0 = 3.7
recalculate predictions, loss, and gradient; repeat

A learning rate that is too small makes progress painfully slow. One that is too large can jump across the valley, oscillate, or diverge toward worse loss. Training tracks loss on separate validation data because lower training loss alone may eventually mean memorization rather than better generalization.

Scenario: Training loss becomes 2.1, 1.4, 3.8, 12.0, 90.0. The exploding sequence suggests steps are unstable, often from an excessive learning rate or badly scaled inputs.

Tip: Plot loss by iteration. Optimization failures become much easier to distinguish from data and evaluation failures when the learning curve is visible.