ROC, Precision-Recall, and AUC

Evaluate ranking across thresholds

A receiver operating characteristic (ROC) curve plots true-positive rate (recall) against false-positive rate across every threshold. ROC AUC, the area under that curve, can be interpreted as the probability a randomly chosen positive receives a higher score than a randomly chosen negative. An AUC of 0.5 is random ranking; 1.0 is perfect ranking.

A precision-recall (PR) curve plots precision against recall and focuses on positive-class performance. With rare positives, ROC can look strong because vast numbers of true negatives keep false-positive rate numerically small, while precision reveals that most alerts are still wrong. Average precision summarizes the PR curve.

from sklearn.metrics import roc_auc_score, average_precision_score
risk = model.predict_proba(X_test)[:, 1]
print(roc_auc_score(y_test, risk))
print(average_precision_score(y_test, risk))

Warning: AUC evaluates ordering across thresholds, not the quality of one deployed threshold or the calibration of probabilities. A high-AUC model can still be unusable at your allowed false-alarm budget.

Tip: For imbalanced problems, show PR performance beside ROC and compare average precision with the positive-class prevalence baseline.