如何判定心跳二分类器的准确率具备统计显著性？

Hey there! Let's break down how to check if your custom heartbeat binary classifier (normal/abnormal) has statistically significant results, especially since you hit a snag with Ébe Isaac's t-test approach. Here's a practical, step-by-step guide tailored to your situation:

You’re aiming to confirm either:

• Your model’s accuracy isn’t just a random fluke (vs. a baseline like random guessing)
• Your model performs significantly better/worse than Model A (since you have its confusion matrix)
  You’ve already set α=0.1, which means we’re looking for a p-value < 0.1 to call the result statistically significant.

First, pull these values from both your model’s confusion matrix and Model A’s:

• True Positives (TP), True Negatives (TN), False Positives (FP), False Negatives (FN)
• Calculate accuracy for each model:
  Accuracy = (TP + TN) / (TP + TN + FP + FN)

Most folks get stuck here for one of these reasons—let’s address them:

Problem 1: You only have a single accuracy value (not multiple samples) #

T-tests rely on having a distribution of accuracy scores, not just one number. To fix this:

• Use k-fold cross-validation: Split your dataset into k equal parts (e.g., 10 folds). Train your model on k-1 folds, test on the remaining 1, and repeat k times. You’ll end up with k accuracy scores for your model and k for Model A.
• Or use bootstrap resampling: From your test set, randomly sample (with replacement) the same number of samples 1000+ times. Calculate accuracy each time to build a distribution of scores.

Problem 2: You’re not sure which t-test to use #

• If you’re comparing your model to random guessing (50% accuracy for binary classification): Use a one-sample t-test. Compare your distribution of accuracy scores to the 50% baseline.
• If you’re comparing your model to Model A: Use a paired t-test (since both models are tested on the same folds/samples, the data is paired). Calculate the difference between each pair of accuracy scores (your model’s score minus Model A’s), then test if the mean difference is significantly different from 0.

Let’s say you did 10-fold cross-validation:

1. Collect your accuracy arrays:

   acc_my_model = [0.89, 0.91, 0.87, 0.90, 0.88, 0.92, 0.86, 0.90, 0.89, 0.91]
   acc_model_A = [0.85, 0.87, 0.83, 0.86, 0.84, 0.88, 0.82, 0.85, 0.84, 0.86]
   

2. Calculate the differences between each fold’s scores:

   differences = [my_acc - a_acc for my_acc, a_acc in zip(acc_my_model, acc_model_A)]
   

3. Use a stats library (like scipy) to run the paired t-test:

   from scipy.stats import ttest_rel
   t_stat, p_value = ttest_rel(acc_my_model, acc_model_A)
   

4. Check the p-value: If p_value < 0.1, your model’s performance is statistically significantly different from Model A.

• Don’t ignore class imbalance: If normal heartbeats are way more common than abnormal ones, accuracy alone can be misleading. Pair your significance test with metrics like precision, recall, or F1-score to get the full picture.
• Small sample sizes? Go non-parametric: If you can’t get enough folds/samples (e.g., k < 5), swap the t-test for the Wilcoxon Signed-Rank Test—it works better with small, non-normal datasets.
• No data leakage!: Make sure your test folds/samples are never used for training—this will skew your accuracy scores and make the significance test meaningless.

内容的提问来源于stack exchange，提问作者Sreehari R