Accuracy, Precision, Recall, and F1

Different mistakes have different costs

To talk about a model's mistakes precisely, it helps to name the four possible outcomes for a yes/no prediction. A true positive (TP) is a correctly caught positive case (the model said "will cancel" and the customer did). A true negative (TN) is a correctly identified negative case. A false positive (FP) is a false alarm - the model raised a flag that turned out to be wrong. A false negative (FN) is a miss - a real positive case the model failed to catch.

Accuracy is simply (TP + TN) / everything, and it's the metric most beginners reach for first - but it can badly mislead on imbalanced data. Recall the earlier example: if only 18% of customers ever cancel, a model that predicts "never cancels" for everyone is 82% accurate while catching precisely zero real churners. That number looks impressive and is completely useless.

Two metrics dig deeper. Precision is TP / (TP + FP) and answers: when the model raises an alarm, how often is it actually right? Recall is TP / (TP + FN) and answers: out of all the real positive cases that existed, how many did the model actually find? These two usually trade off against each other - casting a wider net (raising more alarms) tends to catch more real cases (higher recall) at the cost of more false alarms (lower precision).

from sklearn.metrics import classification_report
print(classification_report(y_test, predictions, target_names=["stayed", "cancelled"]))

              precision    recall  f1-score   support

      stayed       0.87      0.94      0.90       820
   cancelled       0.61      0.41      0.49       180

    accuracy                           0.84      1000

Read that table like this: for customers who actually cancelled, the model's precision is 0.61 (61% of its "will cancel" alarms were correct) and its recall is only 0.41 (it caught just 41% of the customers who really did cancel) - noticeably weaker than its performance on the majority "stayed" class, even though overall accuracy (0.84) looks reasonably good.

The F1 score is the harmonic mean of precision and recall - a single number that rewards a balance between the two, rather than doing well on one while ignoring the other. It's convenient for quick comparisons, but it still hides which kind of error is more expensive for your specific use case, so it should rarely be the only number you look at.

Scenario: A hospital builds a model to flag patients for a disease-screening follow-up and selects the version with the highest accuracy. That model turns out to miss more sick patients (low recall) while being extremely confident about healthy ones. The team switches its selection metric to recall - accepting some extra unnecessary follow-up appointments (lower precision) in exchange for catching far more real cases. For a review queue where each false alarm costs a human reviewer's time, the right trade-off often runs the other way, favoring precision.

Goal: In the model-evaluation lab, you'll repair a broken pipeline, compare a dummy baseline against logistic regression, and select the winning model using the operational metric the lab actually asks for - not whichever number happens to look biggest.