Adam Adapts Each Parameter's Step

Momentum plus scale-aware updates

Adam, short for adaptive moment estimation, tracks moving averages of both gradients and squared gradients. The first behaves like momentum; the second estimates recent gradient scale. Parameters with consistently large gradients receive normalized steps, while sparse or smaller-gradient parameters can receive relatively larger attention.

Adam often works well with little tuning and is a common starting optimizer for deep networks and Transformers. It is not universally superior: SGD with momentum can generalize as well or better on some tasks, and Adam still needs a sensible learning rate, schedule, and regularization.

SGD:  one global learning-rate scale
Adam: global rate adjusted by each parameter's gradient history

Tip: Treat optimizer choice as an experiment under a fixed data split and budget. Record learning curves, final validation quality, steps, and wall-clock time.

Note: A learning-rate schedule changes the rate over training - often warming up gently, then decaying - because early exploration and late fine adjustment benefit from different step sizes.