Precision, recall, and F1 score are statistics used to judge how well a classification model finds the cases we care about. They are especially important when the classes are imbalanced, such as rare diseases, fraud, or equipment failures. In these situations, accuracy can look high even when the model misses most of the important positive cases.
These metrics focus attention on different kinds of mistakes and their real costs.
A confusion matrix sorts predictions into true positives, false positives, true negatives, and false negatives. Precision asks how many predicted positives were actually positive, while recall asks how many actual positives were found. The F1 score combines precision and recall into one value using the harmonic mean, which strongly penalizes a model if either one is low.
Changing the decision threshold usually creates a tradeoff: raising recall often lowers precision, and raising precision often lowers recall.
Understanding Statistics: Precision, Recall, and F1 Score
A classifier often produces a score before it produces a final label. A hospital model might assign a patient a risk score between zero and one for a disease. A chosen cutoff turns that score into a positive or negative result.
If the cutoff is lowered, more patients are flagged. This catches more real cases, yet it creates more unnecessary alerts. If the cutoff is raised, fewer patients are flagged.
The alerts become more trustworthy, but some real cases are missed. The best cutoff depends on what happens after a result, not on a universal rule.
The cost of each error is different in different settings. For a first screening test for a serious illness, missing a patient can be dangerous. High recall is often preferred because flagged patients can receive a second, more accurate test.
For an email spam filter, wrongly hiding an important message is frustrating. In that case, high precision may matter more. Fraud systems face both costs.
Missed fraud loses money, while falsely declined card payments inconvenience customers. People designing a model need to state whose mistakes matter and what action follows each prediction.
The F1 score is useful when both kinds of positive-class performance matter and one low result should not be hidden by one high result. Its harmonic mean gives more weight to the weaker of precision and recall than an ordinary average would. Still, F1 does not include true negatives.
That is usually sensible when negatives are very common, but it can be incomplete in other tasks. It treats false positives and false negatives as equally serious. A weighted version, called an F beta score, can place greater emphasis on recall or precision when the real costs are unequal.
Students should examine the raw counts behind every score. A precision value based on ten predicted positives is much less reliable than the same value based on ten thousand predictions. Results can change across schools, hospitals, neighborhoods, or time periods because the data source changes.
A model should be tested on new data that was not used to train or tune it. It is important to check whether the positive label was assigned correctly, since unclear labels can make every metric misleading.
A precision recall curve compares many cutoffs and shows the available tradeoff. This helps people choose a threshold deliberately instead of reporting one attractive number.
Key Facts
- Precision = TP / (TP + FP)
- Recall = TP / (TP + FN)
- F1 score = 2PR / (P + R), where P is precision and R is recall
- Accuracy = (TP + TN) / (TP + FP + TN + FN)
- False positive means the model predicted positive, but the true class was negative.
- False negative means the model predicted negative, but the true class was positive.
Vocabulary
- Confusion matrix
- A table that counts correct and incorrect predictions for each actual class.
- Precision
- The fraction of positive predictions that are actually correct.
- Recall
- The fraction of actual positive cases that the model correctly finds.
- F1 score
- A single score that combines precision and recall using the harmonic mean.
- Decision threshold
- The cutoff score a model uses to decide whether to label a case as positive.
Common Mistakes to Avoid
- Using accuracy alone on imbalanced data is misleading because a model can get a high accuracy by mostly predicting the majority class.
- Confusing precision with recall is wrong because precision measures correctness among predicted positives, while recall measures coverage of actual positives.
- Ignoring false negatives is dangerous when missing a positive case has a high cost, such as failing to detect a disease.
- Averaging precision and recall with the ordinary mean is not the F1 score because F1 uses the harmonic mean, which penalizes an uneven balance.
Practice Questions
- 1 A model has TP = 40, FP = 10, TN = 900, and FN = 50. Calculate precision, recall, F1 score, and accuracy.
- 2 A spam filter makes 120 positive predictions, and 96 of them are truly spam. There are 150 spam emails total. Find the precision and recall.
- 3 For a medical screening test, explain whether you would prefer higher recall or higher precision, and describe the type of mistake you are trying hardest to avoid.