Diagnose Before Retraining

Connect symptoms to evidence

When something feels off in production, the first and most-skipped step is to confirm the evaluation is actually looking at production data - not accidentally re-scoring the training set, which would hide the whole problem. Once that's verified, look for which features moved (from the previous section) and which confusion-matrix cell grew (from two sections back), then slice the results by time, region, device type, or customer segment to see whether the failure is spread evenly or concentrated in one specific slice - a concentrated failure is usually far easier to explain and fix.

Scenario: A default-risk detector still reports excellent accuracy on its training data, but production customers now skew toward much longer subscription tenures than the training set ever saw. A confusion matrix computed on fresh production outcomes shows a spike in missed defaults, and a distribution check confirms tenure_months has shifted well outside its training range - together, the two pieces of evidence point at the same root cause.

Once you've localized the problem, several responses are available, and "immediately retrain" is rarely the first or best one: repair an upstream data transform that's silently broken, adjust the model's decision threshold if the trade-off between precision and recall has simply shifted, deliberately collect fresh, representative labeled examples, retrain on approved recent data once you have enough of it, or temporarily route the most uncertain predictions to a human reviewer while you investigate further. Whatever you choose, validate the fix on a later holdout of data the fix wasn't tuned against before shipping it.

Sustainable monitoring needs a few concrete pieces in place: a fixed reference window to compare against, a rolling current window, explicit alert thresholds, a named owner, and a written runbook describing what to do when an alert fires. Always show raw counts alongside percentages - a dramatic-sounding "200% increase in false negatives" is a lot less alarming once you learn it went from 3 cases to 9.

Goal: In the drift lab, you'll fix a report that's evaluating the wrong dataset and use both error counts (the confusion matrix) and distribution statistics (feature shift) together to support your diagnosis - exactly the two-pronged approach described above.