Two Ways a Model Fails

Too simple or too attached to training data

Underfitting occurs when a model cannot capture useful structure: training and validation performance are both poor. Overfitting occurs when it captures training-specific noise or shortcuts: training performance is excellent while validation performance lags. The desired property is generalization, useful performance on unseen cases from the deployment population.

model           train error   validation error
too simple         high             high
useful capacity    low              low
too complex        tiny             high

Model capacity describes how complex a pattern it can represent. Tree depth, polynomial degree, neighbor count, feature count, and neural-network size all influence capacity. More capacity reduces training error but does not guarantee lower unseen error.

Analogy: An underfitting student learned only one rule for every problem. An overfitting student memorized the practice answers. The prepared student learned principles that transfer to unfamiliar questions.

Warning: A small train-validation gap does not prove success if both scores are poor. Look at absolute quality against the baseline as well as the gap.