Pay for Complexity

Fit the data without extreme weights

Regularization adds a penalty for model complexity to the training objective. For linear models, complexity is commonly represented by coefficient size. Training now balances prediction loss against the penalty, discouraging a model from relying on extreme weights that may fit noise.

ordinary objective   = prediction loss
regularized objective= prediction loss + strength × weight penalty

Regularization introduces some bias intentionally to reduce variance. The right strength is chosen through validation. In scikit-learn logistic regression, C is inverse strength: smaller C means stronger regularization, an easy direction to remember incorrectly.

Analogy: Packing every possible tool makes a suitcase perfectly prepared for the training trip but heavy and fragile. A baggage fee forces a choice: carry only tools useful enough to justify their cost.

Warning: Features must be comparably scaled before coefficient penalties are interpreted fairly. Otherwise a feature's unit changes how large its coefficient needs to be.