Build an Honest Holdout Test

Reserve unseen evidence

A train/test split reserves one part of the dataset for final evaluation. train_test_split shuffles and divides rows. random_state=42 makes that random division reproducible so teammates investigate the same evidence.

A test set that is tiny produces an unstable score: one row can change the result dramatically. A common starting point is 20% to 30%, adjusted for dataset size and the number of rare examples.

Stratification preserves class proportions in both partitions. Use stratify=y for a classification split when possible. For time-dependent data, random splitting may be wrong; train on earlier records and test on later ones instead.

Never tune repeatedly against the final test set. Use cross-validation or a separate validation set for model choices, then use the held-out test once for an unbiased estimate.

Warning: Preprocessing the full dataset before splitting can leak test-set statistics into training. Put learned preprocessing inside a pipeline.