Local Minima and Practical Optimization

Downhill does not promise the best valley

A local minimum is lower than nearby points but not necessarily the lowest point everywhere. A global minimum is the lowest possible loss across the full parameter space. Convex objectives such as ordinary linear regression have a bowl-like structure where local and global minima coincide; neural networks have far more complicated surfaces.

Modern training succeeds not because gradients guarantee a perfect global solution, but because large models, good initialization, mini-batches, momentum, adaptive optimizers, normalization, and extensive empirical practice often find useful low-loss regions. The neural-network optimizer module later expands those techniques.

start A -> shallow valley (useful solution)
start B -> deeper valley  (different useful solution)
objective: generalize well, not win an abstract lowest-training-loss contest

Warning: Optimization and generalization are different. A model can reach extremely low training loss by memorizing data and still perform poorly on new cases.

Analogy: Reaching the bottom of the nearest valley solves navigation locally. Choosing whether that valley leads toward the right city is a separate evaluation problem.