Choose Thresholds and Check Calibration

Ranking, decisions, and probability meaning differ

A threshold converts scores to actions. Choose it on validation data using explicit costs, capacity, or constraints - for example, maximize recall while keeping precision above 60%, or review only the 200 highest-risk cases per day. Then report test performance at the frozen threshold.

Calibration asks whether predicted probabilities match observed frequencies. Among cases scored near 0.7, roughly 70% should be positive if the score is well calibrated. A calibration curve compares predicted and observed rates; the Brier score averages squared probability error.

ranking question: are positives ordered above negatives?
threshold question: which cases trigger action?
calibration question: does 0.7 really mean about 70%?

Scenario: Two models share the same ROC AUC. One's 0.8 bucket contains 80% positives; the other's contains 45%. Both rank similarly, but only the first supports literal risk communication without recalibration.

Warning: Recalibrate and reconsider thresholds when prevalence or population changes. A probability mapping learned in one environment may not transfer unchanged.