Batch, Stochastic, and Mini-Batch

How much data informs one step?

Batch gradient descent calculates one update from the entire training set: stable but expensive for large data. Stochastic gradient descent (SGD) updates from one example at a time: cheap and noisy. Mini-batch gradient descent uses a moderate group, balancing efficient matrix computation with frequent, somewhat noisy updates.

full batch: 10,000 examples -> 1 smooth update per epoch
stochastic: 1 example -> 10,000 noisy updates per epoch
mini-batch: 100 examples -> 100 updates per epoch

Noise is not purely harmful; it can shake optimization out of narrow regions. But tiny batches may produce unstable directions, while very large batches consume memory and may require learning-rate adjustment.

Analogy: Decide a restaurant menu from every customer's feedback at once, one customer at a time, or daily groups. Full consensus is slow; single opinions swing wildly; groups offer a workable rhythm.

Tip: Report batch size with learning rate and optimizer. These settings interact, so comparing one while silently changing another is not controlled evidence.