WEKA中的Kappa Statistic：是否为标注者间一致性指标，与Cohen/Fleiss Kappa相似？

Great question! Let's break this down clearly, since WEKA's Kappa implementations can be a bit confusing depending on the use case.

First: Is WEKA's Kappa an inter-rater reliability metric? #

Short answer: Yes—but it depends on how you're using it. At its core, WEKA's Kappa Statistic falls into the inter-rater reliability (IRR) family of metrics, which measure agreement between multiple "judges" (or in ML terms, models vs. human labels, or multiple human annotators).

How does it compare to Cohen's Kappa and Fleiss Kappa? #

Let's break down the similarities and key differences:

1. Alignment with Cohen's Kappa #

• Cohen's Kappa is designed specifically for two raters (e.g., two human annotators, or a model's predictions vs. a single human's gold labels) assessing the same set of categorical samples, correcting for agreement that would happen by random chance.
• In WEKA, when you calculate Kappa via the standard Evaluation class for a classification model (comparing model predictions to a single gold label set), you're effectively computing Cohen's Kappa. This treats the model as one "rater" and the gold labels as the second. If you have two human annotator sets and use WEKA to compare them, you'll also get Cohen's Kappa.

2. Alignment with Fleiss Kappa #

• Fleiss Kappa is built for three or more raters, measuring overall agreement across all annotators for a set of categorical samples (again, correcting for random chance).
• WEKA has a dedicated class—weka.stats.InterRaterAgreement—that implements Fleiss Kappa. If you're working with data from three+ human annotators, this is the tool you'll use to calculate the multi-rater agreement, which matches exactly what Fleiss Kappa does.

A common point of confusion #

Don't mix up model evaluation Kappa with multi-annotator IRR:

• When you use WEKA to evaluate a classifier, Kappa is measuring agreement between the model and gold labels (two "raters").
• When you use InterRaterAgreement, you're measuring agreement between multiple human annotators—this is the true multi-rater IRR use case for Fleiss Kappa.

内容的提问来源于stack exchange，提问作者PaulJVC