机器学习中测试集过小的影响及小比例测试集的可行性分析

Great question—this is a critical concern in medical machine learning, where data scarcity is often a real, unavoidable constraint. Let’s break down both the feasibility of your specific case and the key risks of using a small test set.

First: Is this 2.5% test set feasible? #

It depends on a few core factors:

• Data homogeneity & class distribution: If your 1500 patients are highly similar (e.g., same disease subtype, consistent demographic group) and the task is binary with roughly balanced classes (so ~18 samples per class in the test set), this might be just barely acceptable for a preliminary evaluation. But if your data is heterogeneous (mixed demographics, varying disease severities) or it’s a multi-class task, 36 samples likely won’t capture the full range of cases your model will face in real clinical settings.
• Role of cross-validation: Since you’re using 5-fold cross-validation alongside an independent test set, the CV results will do most of the heavy lifting in estimating generalization. The test set here acts as a final "sanity check" to confirm CV isn’t overfitting to training folds. If your CV scores are consistent and the test set results align closely, this setup might be defensible—especially if collecting more patient data is logistically impossible (a common hurdle in medical research).
• Medical domain context: For rare diseases or niche patient populations, small test sets are often the only option. In these cases, you’ll need to pair your performance metrics with confidence intervals (e.g., using bootstrap resampling) to show the uncertainty in your results, rather than reporting a single accuracy number.

Key risks of using a small test set #

Even if your setup is defensible, small test sets introduce several meaningful problems:

• Unstable performance estimates: A single misclassified sample can swing your accuracy by ~2.8% (1/36). If your test set happens to include an unusual cluster of easy or hard cases, your results won’t reflect the model’s true real-world performance.
• Low statistical power: You won’t be able to reliably detect meaningful differences between models (e.g., KNN vs. SVM). A small sample size means even a true performance gap might not show up as statistically significant, leading you to make wrong decisions about which model to deploy.
• Exacerbated class imbalance: If one class is already underrepresented in your full dataset, the test set might end up with only a handful of samples from that class. For example, if a rare condition makes up 5% of your data, the test set might have only 1-2 samples of it—making it impossible to evaluate how well your model handles that critical subgroup.
• Missing edge cases: Medical data is full of high-stakes edge cases (e.g., patients with comorbidities, atypical symptoms). A small test set is unlikely to include these, so your model’s performance on these scenarios will remain unknown.

Quick actionable recommendations #

If you have flexibility:

• Use stratified sampling to ensure the test set mirrors the class distribution of your full dataset.
• Consider increasing the test set size to at least 5-10% of your data (75-150 samples) if possible—this will drastically improve the stability of your estimates.
• Report confidence intervals for your test set metrics (using bootstrap or binomial methods) to communicate the uncertainty in your results clearly.

内容的提问来源于stack exchange，提问作者Kuni