Build an Honest Holdout Test

Reserve unseen evidence

If you train a model on every row you have and then check its accuracy on those same rows, you're grading an exam using the study guide as the answer key - the model can partly just be memorizing rather than generalizing. The standard fix is a train/test split: set aside a portion of the dataset that the model never sees during training, and use it only at the very end to get an honest score.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
print(len(X_train), len(X_test))

4000 1000

test_size=0.2 reserves 20% of the rows purely for evaluation. random_state=42 fixes the random shuffling so that running the exact same code twice produces the exact same split - without it, every teammate (and every re-run) would get a slightly different, non-comparable result. A commonly used starting point is a 20-30% holdout, adjusted up for a small dataset (you need enough test rows to trust the score) or down for a very large one.

stratify=y is doing something specific and important: it makes sure both the training set and the test set keep roughly the same proportion of each class as the full dataset. Without it, an unlucky random split could put almost none of the rare "cancelled" customers into the test set, making the test score nearly meaningless for the exact question you care about.

Warning: A very small test set produces an unstable score - flipping the correct prediction on just one or two rows can swing accuracy by several percentage points. If your dataset is small, cross-validation (repeating the split several times and averaging) gives a far more trustworthy number than a single split.

One more subtlety: if your rows are time-ordered (transactions, sensor readings, log events), a random split can accidentally let the model "see the future" - training on some rows from next week and testing on rows from last week. For time-dependent data, split chronologically instead: train on earlier records, test on strictly later ones, to mimic how the model will actually be used.

Never repeatedly tweak your model while checking its score against the same final test set - that quietly turns the test set into another training set, just used less obviously. Use cross-validation or a separate validation set while you're experimenting, and touch the true held-out test set only once, at the very end, to report the number you'll actually trust.

Warning: Preprocessing the entire dataset (like fitting StandardScaler on everything) before splitting is a subtle form of leakage - test-set statistics (its mean, its range) sneak into the numbers the model trains on. Always put preprocessing inside a Pipeline so it's fit on the training fold only, exactly as shown in the previous section.