When Production Data Moves

The world changes after training

A distribution describes how values and outcomes tend to occur in a dataset - the typical range of ages, the usual mix of subscription plans, how often customers actually cancel. Distribution drift means the data your model sees after deployment no longer looks like the data it was trained on. Customer behavior shifts, a sensor gets recalibrated, a pricing policy changes, a new season begins, or the product itself changes - any of these can quietly move the distribution underneath a model that was trained on an older snapshot of the world.

A simple, cheap first diagnostic compares one feature's average in production against its average during training, standardized by how much that feature naturally varies:

shift = abs(prod_mean - train_mean) / train_std
print(round(shift, 2))

1.8

A value like 1.8 means production's average tenure has moved almost two standard deviations away from what training saw - worth investigating, but not proof on its own that the model is now wrong. This kind of statistic is a smoke alarm, not a fire report: it can miss a change in the shape of a distribution (say, a new second cluster of very short-tenure customers) that leaves the overall mean almost unchanged.

Beyond the mean, it's worth comparing missing-value rates (did a source system start failing silently?), quantiles (did the whole spread shift, not just the center?), category frequencies (did one subscription plan suddenly become far more common?), and target prevalence (are more or fewer customers actually cancelling now, independent of the model?). Detecting drift in the inputs is possible even before you know whether predictions are correct; confirming drift actually hurt accuracy requires waiting for real outcomes and recomputing a confusion matrix on fresh, labeled production data.

Warning: A moved statistic is a prompt to investigate, not an automatic order to retrain. Before touching the model, rule out a broken data pipeline, a redefined field (did "income" start being reported monthly instead of yearly?), an unusually small or unrepresentative sample, or a genuine, expected business change like a marketing campaign that temporarily attracted a different kind of customer.