使用公开数据集撰写学术论文，能否采用不同验证方法与其他文献对比？

Hey, this is such a common (and totally valid) question when working with benchmark datasets—let’s break this down clearly:

First, the key risk of direct comparison #

You can’t just take a single 70/15/15 train-test-split accuracy and directly compare it to results from 10-fold cross-validation (CV). Here’s why:

• 10-fold CV averages results across 10 different train/test splits, which reduces random variance and gives a more stable, reliable estimate of model performance.
• A single train-test-split’s accuracy can vary a lot depending on how the data is randomly divided—your 70/15/15 split might by chance have easier/harder samples than the folds used in other studies.
• Reviewers will almost certainly flag this as a flaw in your comparison, since the two methods aren’t apples-to-apples.

The best practice: Align with the dataset’s standard method #

Since the dataset authors and a subset of researchers use 10-fold CV, this should be your go-to approach. Here’s why it matters:

• It ensures your results are directly comparable to the most authoritative work on the dataset (the authors’ own benchmarks).
• It eliminates the "random split bias" issue mentioned above, making your findings more credible.
• For studies that don’t specify their validation method, you can note in your discussion section: "We adopted the dataset authors’ recommended 10-fold cross-validation to maintain consistency with established benchmarks, ensuring our results are comparable to prior work [X, Y]."

If you must use a fixed train/val/test split #

Maybe your model requires a fixed validation set for hyperparameter tuning, or you have another specific reason to use 70/15/15. If so, take these steps to make your comparison as valid as possible:

• Fix the random seed: Use a consistent seed (like random_state=42 in scikit-learn) so your split is fully reproducible. State this explicitly in your paper’s experimental section.
• Repeat the split multiple times: Don’t rely on one split—run the experiment 10-20 times with different random seeds, then report the mean accuracy and standard deviation. This mimics the stability of 10-fold CV and gives reviewers a sense of your model’s performance consistency.
• Be transparent about the difference: In your results section, clearly state that you used a repeated train-test-split, and when comparing to 10-fold CV results, note that the validation strategies differ. For example: "While prior work [X] uses 10-fold CV, we report results from 10 repeated 70/15/15 splits to enable hyperparameter tuning. Our mean accuracy of X% is comparable to [X]’s X% 10-fold CV result, indicating consistent performance across validation strategies."

Bonus: Strengthen your paper with both methods #

If you have the computational resources, run both 10-fold CV and repeated train-test-split experiments. This lets you:

1. Align with standard benchmarks for direct comparison.
2. Show that your model performs consistently across different validation strategies, which adds robustness to your findings.

内容的提问来源于stack exchange，提问作者ohdoughnut