Bagging Builds Independent Opinions

Average many noisy learners

Bagging, short for bootstrap aggregating, trains multiple models independently on bootstrap samples - datasets drawn from the training rows with replacement. Classification combines votes; regression averages predictions. Individual deep trees vary strongly, but their uncorrelated errors partially cancel when averaged.

tree 1 trained on bootstrap sample A -> vote fail
tree 2 trained on bootstrap sample B -> vote safe
tree 3 trained on bootstrap sample C -> vote fail
ensemble majority -> fail

Because sampling uses replacement, some rows appear multiple times and roughly a third are absent from any one sample. Those absent out-of-bag rows can provide an internal validation estimate, though a deliberately designed holdout remains important.

Analogy: Ask many inspectors who each examined a different resampled casebook. If their mistakes are not identical, a majority vote is steadier than trusting one inspector.

Warning: Averaging helps variance, not shared bias. If every tree learns the same leaked feature, the forest becomes confidently wrong together.