Split According to Reality

Random is not always honest

A random split works when rows are independent and drawn from the same process. Stratification preserves the proportion of target classes, preventing a rare class from nearly disappearing from one split. A group split keeps all related rows - every visit from one patient or every event from one server - on one side, preventing identity-specific patterns from leaking across the boundary.

Time-dependent problems require a chronological split: train on the past, validate on a later period, and test on the latest period. Randomly mixing future events into training creates temporal leakage because deployment cannot learn from tomorrow before predicting today.

from sklearn.model_selection import GroupShuffleSplit
splitter = GroupShuffleSplit(test_size=0.2, random_state=42)
train_idx, test_idx = next(splitter.split(X, y, groups=df['server_id']))

Scenario: A fault model has ten rows per server. A random split places the same servers in train and test, so the model recognizes server-specific baselines. It fails on newly installed servers. Grouping by server_id exposes the honest difficulty.

Tip: Design the split to imitate the future handoff: new people, later dates, unseen sites, or whichever boundary production must truly cross.