Diagnose Before Retraining

Connect symptoms to evidence

First confirm the report evaluates production rows rather than accidentally reusing training data. Then locate which features moved and which confusion-matrix cell grew. Slice results by time, region, device, or customer group to find a concentrated failure.

Responses include repairing an upstream transform, adjusting a decision threshold, collecting representative labels, retraining on approved recent data, or temporarily routing uncertain cases to a person. Validate the candidate fix on a later holdout before deployment.

Monitoring needs a reference window, current window, thresholds, ownership, and a response runbook. Keep raw counts beside ratios so a dramatic percentage from three examples does not trigger panic.

Scenario: A default detector still reports excellent training accuracy, but production customers now have longer tenure. A production confusion matrix exposes missed defaults and feature statistics show the shifted population.

Goal: In the drift lab, fix the evaluated dataset and use both error counts and distribution statistics to support the diagnosis.