Learn from Earlier Mistakes

A sequence of small corrections

Boosting trains weak learners sequentially. Each new learner focuses on errors left by the current ensemble, and its output is added as a small correction. Gradient boosting chooses corrections that move predictions downhill on a chosen loss function, connecting directly to gradient-descent intuition.

initial guess: every delivery takes 30 minutes
tree 1 correction: rural routes need +8
tree 2 correction: priority shipping needs -5
tree 3 correction: rain plus traffic needs +6
final = initial + small learned corrections

The learning rate shrinks each tree's contribution. Smaller rates usually require more trees but can generalize better. Tree depth controls interaction complexity; boosted trees are often shallow so each contributes a restrained rule.

from sklearn.ensemble import HistGradientBoostingClassifier
boost = HistGradientBoostingClassifier(
    learning_rate=0.05, max_iter=200, max_leaf_nodes=15, random_state=42
)

Analogy: An editor reads a draft, marks remaining mistakes, and gives the next editor that corrected draft. Each pass focuses on what earlier passes still missed.

Warning: Because each learner depends on the current ensemble, shuffled training order is not interchangeable with bagging's independent parallel learners.