Tune Without Fooling Yourself

Hyperparameters are learned decisions too

A hyperparameter search compares settings such as tree depth, regularization strength, or neighbor count using validation folds. Grid search tries declared combinations; randomized search samples from ranges and can cover large spaces more efficiently. Put preprocessing inside the searched pipeline.

from sklearn.model_selection import GridSearchCV
search = GridSearchCV(
    pipeline, {'model__max_depth': [3, 6, None]},
    scoring='neg_mean_absolute_error', cv=5,
)
search.fit(X_train, y_train)
final_test_pred = search.best_estimator_.predict(X_test)

The best cross-validation score is optimistically biased because it won among many trials. For high-stakes comparison, nested cross-validation uses inner folds for tuning and outer folds for evaluation. For ordinary work, keep a final untouched test set and report the complete search space and selected configuration.

Scenario: A team tries 500 configurations against the same test set and publishes the winner. The test set has become training feedback; its winning score no longer estimates an untouched future.

Note: Prefer the simplest model whose uncertainty interval and operational constraints meet the goal. Tiny metric gains may not justify slower inference, greater instability, or weaker explanations.