When Production Data Moves

The world changes after training

A data distribution describes how values and classes occur. Distribution drift means production inputs or outcomes differ from the data used for training. Customer behavior, sensor calibration, policy, season, and product design can all cause drift.

A simple diagnostic compares a feature's training mean with its production mean. Dividing the absolute difference by the training standard deviation creates a standardized mean shift. It is an alerting clue, not proof of model failure, and it can miss changes in shape that keep the same mean.

Also compare missing rates, quantiles, category frequencies, and target prevalence. Drift without labels can warn that inputs changed. Once delayed labels arrive, calculate confusion matrices and task metrics to see whether correctness changed.

Warning: Do not immediately retrain because one drift statistic moved. Check data-pipeline bugs, measurement definitions, sample size, and business changes first.