Cross-Validation Reduces Split Luck

Rotate the validation role

In k-fold cross-validation, training data is divided into k folds. Train on k-1, validate on the remaining fold, and repeat until every fold served as validation. The mean estimates expected performance; the spread reveals instability. The final test set stays untouched.

from sklearn.model_selection import cross_validate
scores = cross_validate(
    pipeline, X_train, y_train, cv=5,
    scoring=['neg_mean_absolute_error', 'neg_root_mean_squared_error'],
    return_train_score=True,
)

Use stratified folds for imbalanced classification, group folds for related rows, and forward-chaining time splits for temporal data. Ordinary random k-fold can leak users, machines, or future information exactly like an ordinary random holdout.

Analogy: One audition room may flatter or hurt a performer. Rotating through several rooms gives a steadier estimate and reveals sensitivity to conditions.

Warning: Cross-validation does not fix an unrepresentative dataset or leaked features. It repeats the evaluation design you give it, including its mistakes.