Audit the Boundary

Verify rather than assume

After splitting, record row counts, class proportions, date ranges, and group overlap. Check identifiers for accidental duplication and confirm transformation objects have not seen validation or test rows. A fixed random seed makes a randomized split reproducible, but it does not make a poor splitting strategy valid.

assert set(train.server_id).isdisjoint(set(test.server_id))
assert train.timestamp.max() < test.timestamp.min()  # for a time split
print(train.target.value_counts(normalize=True))
print(test.target.value_counts(normalize=True))

A split is part of the experiment definition. Version either the row IDs or the deterministic rule that creates them. Otherwise two people may report different scores while believing they evaluated the same system.

Note: The test set estimates performance only for data resembling its own sampling process. A perfectly sealed but unrepresentative holdout still gives a precise answer to the wrong deployment question.

Goal: Apply stratified splitting in the existing ml-model-evaluation lab and choose a model from the metric the operational scenario actually values.