Leakage and Data Contracts

Do not let the answer sneak into the question

Target leakage is one of the most common and most embarrassing mistakes in applied ML: it happens when a feature contains information that would not actually exist at the moment the prediction is needed, or that directly gives away the target. A model trained on leaked data can look almost magically accurate in testing and then fail completely in the real world, because the shortcut it learned to rely on simply isn't available when it matters.

Scenario: A loan-approval model includes collection_outcome (whether the loan eventually went to debt collection) as a feature. Offline, the model reaches 99% accuracy - because collection_outcome is filled in weeks after the loan is approved, and it almost perfectly determines whether the loan defaulted. In production, at the moment a loan application actually needs a decision, that column doesn't exist yet. Removing it drops the reported accuracy, but that lower number is the honest one - it's what the model can actually achieve with information available at decision time.

A simple, repeatable test for every column is to ask: could the real application know this value at the exact moment it needs to make the prediction? If the answer is no, or "only sometimes," that column is dangerous. Also watch for suspicious correlations that seem too good to be true, duplicated identifiers that let one customer's later behavior leak into their earlier row, columns computed from future rows (like a rolling average calculated over the whole dataset instead of only past data), and any cleaning or scaling that was accidentally done before splitting off a test set.

# A quick leakage smell-test: does a feature correlate almost perfectly
# with the target? That's suspicious, not necessarily good news.
df.corr(numeric_only=True)["cancelled"].sort_values(ascending=False)

cancelled              1.00
exit_survey_score      0.97   # <- huge red flag, investigate this column
tenure_months         -0.31
age                    0.02

A near-perfect correlation like 0.97 between a feature and the target is rarely a lucky discovery - it's usually a sign that the feature was recorded because of the target (an exit survey is only filled out by customers who already cancelled), rather than something known beforehand.

To catch these mistakes automatically instead of relying on memory, teams write a data contract: a small, explicit record of what a clean dataset is supposed to look like - expected column names and types, allowed value ranges, rules for missing values, and a written definition of the target. Turning that contract into code means a corrupted or leaking dataset fails loudly instead of quietly poisoning a model:

assert set(df.columns) >= {"age", "plan", "income", "tenure_months", "cancelled"}
assert df["income"].isna().sum() == 0, "income should be fully imputed by this point"
assert df["age"].between(0, 120).all(), "age has impossible values"

Goal: In the data-hygiene lab, you'll produce exactly the safe numeric features the checker asks for, while keeping every row correctly aligned with its target - the same discipline shown above, applied under time pressure.