From Examples to Batches

Learn from several examples per update

A batch is a group of examples processed together before one parameter update. Vectorized batch computation uses hardware efficiently and averaging reduces the noise of any single example. Batch size changes memory use and gradient behavior, not merely speed.

example losses: 0.2, 0.8, 0.4, 0.6
batch mean loss: (0.2+0.8+0.4+0.6)/4 = 0.5
backpropagate from that aggregate

A training epoch is one pass through the available training examples. Shuffling between epochs prevents a fixed ordering from creating systematic updates, unless sequence or time structure makes shuffling inappropriate.

Scenario: Training examples are sorted by class and processed without shuffling. Long runs of one class push parameters one way, then the next class pushes them back. Mixed mini-batches provide steadier evidence.

Note: Large batches estimate the full-data gradient smoothly but consume memory; small batches provide noisier updates that can help exploration. The optimizer module compares these choices directly.