A Concrete Regularization Comparison

Compare unseen evidence, not coefficient beauty

Suppose a polynomial regression with 30 derived features achieves training root mean squared error (RMSE) of 1.1 and validation RMSE of 8.7. Adding L2 regularization raises training RMSE to 2.4 but lowers validation RMSE to 3.2. The model fits training less perfectly and generalizes far better - the intended trade.

from sklearn.linear_model import Ridge
from sklearn.model_selection import GridSearchCV
grid = GridSearchCV(
    pipeline_with_ridge,
    {'model__alpha': [0.01, 0.1, 1, 10, 100]},
    scoring='neg_root_mean_squared_error', cv=5,
)
grid.fit(X_train, y_train)

Ridge uses alpha, where larger values mean stronger L2 regularization - opposite logistic regression's C. Search plausible values on a logarithmic scale because useful strengths often differ by orders of magnitude.

Tip: Record train and validation metrics, selected strength, coefficient norm, and fold spread. A smaller coefficient norm is not the goal by itself; honest unseen performance is.

Warning: Do not use regularization to mask leakage or a broken split. A penalized leaked feature can remain devastatingly informative.