Fix a classifier with high accuracy that misses the rare class

A classifier reports 99 percent accuracy while failing to detect the cases the team cares about. Most examples belong to the majority class, so a trivial predictor can look impressive.

Why this happens

Accuracy alone hides the distribution of errors. False negatives and false positives can have different operational consequences, and a default decision threshold may not suit the task.

Step-by-step solution

1. Count examples per class and calculate a majority-class baseline. Inspect the confusion matrix rather than relying on one headline metric.

2. Measure precision and recall for the important class. Choose a threshold using validation data and the application’s error costs.

3. Evaluate the final choice on held-out data. Consider class weighting or resampling within training folds, while preserving a realistic evaluation distribution.

Worked example

from sklearn.metrics import confusion_matrix, recall_score
actual = [0,0,0,0,1]
predicted = [0,0,0,0,0]
assert recall_score(actual, predicted, zero_division=0) == 0
assert confusion_matrix(actual, predicted).tolist() == [[4,0],[1,0]]

Verify the fix

The example has four correct predictions but zero recall for class 1. Your dashboard should make that failure visible instead of describing the model as successful.

Prevent a repeat

Monitor class-specific metrics and changes in prevalence. Do not tune the threshold repeatedly on the final test set.

Reference

Official documentation or standards guidance

An original Gistmax troubleshooting guide. Scenarios and sample values are illustrative; configuration depends on your software version and environment. Cover diagram: Gistmax Tutorials.