不平衡数据集下AUC、准确率与F1-score的解读困惑

Great question—this is a super common pitfall when working with imbalanced multiclass datasets, and your confusion comes from not digging into how these metrics are calculated under the hood, especially for multiclass scenarios. Let's break down exactly what's happening with each metric in your case, and what you're missing.

1. 为什么你的所有指标都“虚高”？ #

Let's start with your specific dataset: 950 class 1 samples, 30 class 2, 20 class 3. Your model outputs [[0.7, 0.1, 0.2]] for all samples (effectively predicting class 1 for everyone), and you're seeing high accuracy, AUC, and F1-score. Here's why each metric is misleading:

分类准确率 #

This one's straightforward: accuracy counts the percentage of correct predictions, and since 95% of your data is class 1, predicting class 1 for everyone will naturally give you 95% accuracy. As you noted, this metric is useless for imbalanced data because it rewards ignoring minority classes entirely.

F1-score (sklearn default) #

When you ran sklearn.metrics.f1_score without specifying the average parameter, it defaults to weighted average. This means each class's F1-score is multiplied by the number of samples in that class, then averaged.

• For class 1: All 950 samples are correctly predicted, so F1-score is 1.0.
• For class 2: 0 samples are correctly predicted (all are labeled class 1), so F1-score is 0.
• For class 3: Same as class 2, F1-score is 0.

The weighted average becomes (950*1 + 30*0 + 20*0)/1000 = 0.95, which makes the model look good—but it's just reflecting the dominance of class 1. If you use macro average (average='macro'), which treats all classes equally, you get (1 + 0 + 0)/3 ≈ 0.33, which accurately shows the model's failure on minority classes.

AUC (TF Keras default) #

TF Keras's AUC metric for multiclass uses one-vs-rest (OVR) classification by default, and it also defaults to weighted averaging (weighting each class's AUC by its sample count). Let's break down each class's OVR AUC:

• Class 1 vs rest: All true class 1 samples have a prediction probability of 0.7, which is higher than the 0.1/0.2 probabilities of non-class 1 samples. So this AUC is 1.0.
• Class 2 vs rest: All true class 2 samples have a prediction probability of 0.1, which is lower than the 0.7 (class 1) and 0.2 (class 3) probabilities of other samples. So this AUC is 0.
• Class 3 vs rest: True class 3 samples have a probability of 0.2. Class 1 samples have 0.7 (higher than 0.2), class 2 samples have 0.1 (lower than 0.2). The AUC here is the fraction of correctly ordered pairs: 30/(30+950) ≈ 0.03.

Weighted averaging these gives (950*1 + 30*0 + 20*0.03)/1000 ≈ 0.95, which rounds to your reported 0.96. Again, the majority class is pulling the average up. If you use unweighted macro average AUC in Keras (average='macro'), you'd get (1 + 0 + 0.03)/3 ≈ 0.34, which tells the real story.

2. 你忽略的关键细节 #

The core issue is that you're relying on default metric settings which are optimized for balanced datasets. Here's what you missed:

• Metric averaging matters: Weighted averages prioritize majority classes, while macro averages treat all classes equally. For imbalanced data, macro averages (or per-class metrics) are far more informative.
• Multiclass AUC has multiple flavors: OVR vs one-vs-one (OVO), weighted vs unweighted—each gives a very different picture. Default weighted OVR AUC will always favor majority classes.
• You need to look at per-class metrics: Instead of just overall scores, check the precision, recall, and F1-score for each individual class. This will immediately show you that classes 2 and 3 have 0 recall (the model never correctly identifies them).

3. 正确的指标解读和评估方法 #

To get an accurate picture of your model's performance on imbalanced multiclass data, do these things:

a. 查看每个类别的单独指标 #

Calculate precision, recall, and F1-score for each class individually. For your case, this will show:

• Class 1: Precision=1.0, Recall=1.0, F1=1.0
• Class 2: Precision=0, Recall=0, F1=0
• Class 3: Precision=0, Recall=0, F1=0

This makes it crystal clear that the model is only good at class 1 and fails entirely on the minority classes.

b. 使用对不平衡数据友好的平均方式 #

• Macro average: Use average='macro' for F1-score, precision, recall, or AUC. This treats all classes equally, so minority class performance isn't hidden.
• Avoid weighted/micro averages: Weighted averages favor majority classes; micro averages treat all samples equally (so again, majority classes dominate).

c. 选择更适合不平衡数据的指标 #

• 马修斯相关系数（MCC）: This metric considers all elements of the confusion matrix and is extremely robust to imbalanced data. For your model, MCC will be near 0, which correctly indicates no predictive power beyond guessing the majority class.
• PR AUC (Precision-Recall AUC): ROC AUC (what you calculated) can be misleading for imbalanced data because it focuses on true negative rate. PR AUC focuses on the tradeoff between precision and recall for positive (minority) classes, giving a more accurate view of how well your model identifies rare samples.
• 混淆矩阵: A simple confusion matrix will show you exactly how many samples from each class are being misclassified—no need for complex metrics to see that classes 2 and 3 are all being predicted as class 1.

总结 #

Your confusion comes from using default metric settings that are biased toward majority classes. To properly evaluate imbalanced multiclass models:

• Always check per-class metrics first.
• Use macro-averaged metrics instead of weighted ones.
• Consider metrics like MCC or PR AUC that are designed for imbalanced data.

内容的提问来源于stack exchange，提问作者abhi