为何不同Cross-Validation技术得出完全相同的Evaluation Metrics？

我实现了K-Nearest Neighbor、Decision Trees和Random Forest三种机器学习算法，针对每种算法使用Hold-Out Method、Leave-One-Out Method、K-Fold Cross-Validation、Stratified K-Fold Cross-Validation四种交叉验证技术，目的是评估性能并对比表现。但代码运行后，不同技术得到的评估指标值完全一致，请问这是正常情况还是代码有问题？

部分代码 #

# Initialize classifiers
knn = KNeighborsClassifier(n_neighbors=5, metric='minkowski', p=2)
dtree = DecisionTreeClassifier(random_state=42)
rf = RandomForestClassifier(n_estimators=20, criterion='entropy', random_state=0)

classifiers = {'KNN': knn, 'Decision Tree': dtree, 'Random Forest': rf}

# Define cross-validation methods
loo = LeaveOneOut()
kf = KFold(10)
skf = StratifiedKFold(n_splits=5)

cv_methods = {'Hold-Out Method': (X_train, X_test, y_train, y_test),
              'Leave-One-Out Method': loo,
              'K-Fold Cross-Validation': kf,
              'Stratified K-Fold Cross-Validation': skf}

# Perform classification and evaluation for each classifier and cross-validation method
for clf_name, clf in classifiers.items():
    print(f"Classifier: {clf_name}")
    for cv_name, cv_method in cv_methods.items():
        if cv_name == 'Hold-Out Method':
            X_train_cv, X_test_cv, y_train_cv, y_test_cv = cv_method
            clf.fit(X_train_cv, y_train_cv)
            y_pred = clf.predict(X_test_cv)
        else:
            scores = cross_val_score(clf, X, y, cv=cv_method, scoring='accuracy')
            

        # Calculate evaluation metrics
        accuracy = accuracy_score(y_test_cv, y_pred)
        precision = precision_score(y_test_cv, y_pred, average='weighted')
        recall = recall_score(y_test_cv, y_pred, average='weighted')
        f1 = f1_score(y_test_cv, y_pred, average='weighted')
        confusion = confusion_matrix(y_test_cv, y_pred)

输出结果 #

Classifier: KNN
Hold-Out Method Metrics for KNN:
Accuracy: 0.864620939
Precision: 0.8661
Recall: 0.8646
F1 Score: 0.8652
Confusion Matrix:
[[326  41]
 [ 34 153]]

Leave-One-Out Method Metrics for KNN:
Accuracy: 0.864620939
Precision: 0.8661
Recall: 0.8646
F1 Score: 0.8652
Confusion Matrix:
[[326  41]
 [ 34 153]]

K-Fold Cross-Validation Metrics for KNN:
Accuracy: 0.864620939
Precision: 0.8661
Recall: 0.8646
F1 Score: 0.8652
Confusion Matrix:
[[326  41]
 [ 34 153]]

Stratified K-Fold Cross-Validation Metrics for KNN:
Accuracy: 0.864620939
Precision: 0.8661
Recall: 0.8646
F1 Score: 0.8652
Confusion Matrix:
[[326  41]
 [ 34 153]]

Classifier: Decision Tree
Hold-Out Method Metrics for Decision Tree:
Accuracy: 0.980144404
Precision: 0.9801
Recall: 0.9801
F1 Score: 0.9801
Confusion Matrix:
[[363   4]
 [  7 180]]

Leave-One-Out Method Metrics for Decision Tree:
Accuracy: 0.980144404
Precision: 0.9801
Recall: 0.9801
F1 Score: 0.9801
Confusion Matrix:
[[363   4]
 [  7 180]]

K-Fold Cross-Validation Metrics for Decision Tree:
Accuracy: 0.980144404
Precision: 0.9801
Recall: 0.9801
F1 Score: 0.9801
Confusion Matrix:
[[363   4]
 [  7 180]]

Stratified K-Fold Cross-Validation Metrics for Decision Tree:
Accuracy: 0.980144404
Precision: 0.9801
Recall: 0.9801
F1 Score: 0.9801
Confusion Matrix:
[[363   4]
 [  7 180]]

Classifier: Random Forest
Hold-Out Method Metrics for Random Forest:
Accuracy: 0.981949458
Precision: 0.9820
Recall: 0.9819
F1 Score: 0.9819
Confusion Matrix:
[[364   3]
 [  7 180]]

Leave-One-Out Method Metrics for Random Forest:
Accuracy: 0.981949458
Precision: 0.9820
Recall: 0.9819
F1 Score: 0.9819
Confusion Matrix:
[[364   3]
 [  7 180]]

K-Fold Cross-Validation Metrics for Random Forest:
Accuracy: 0.981949458
Precision: 0.9820
Recall: 0.9819
F1 Score: 0.9819
Confusion Matrix:
[[364   3]
 [  7 180]]

Stratified K-Fold Cross-Validation Metrics for Random Forest:
Accuracy: 0.981949458
Precision: 0.9820
Recall: 0.9819
F1 Score: 0.9819
Confusion Matrix:
[[364   3]
 [  7 180]]

问题原因分析 #

• 交叉验证分支未更新核心变量：在处理Leave-One-Out、K-Fold、Stratified K-Fold的else分支中，你仅调用cross_val_score计算了准确率数组，但没有更新y_test_cv和y_pred变量。后续计算指标时，依然复用了Hold-Out Method分支中赋值的这两个变量，导致所有交叉验证方法都输出了Hold-Out的结果。
• cross_val_score的使用局限：cross_val_score只能返回各折的准确率，无法直接生成用于计算precision、recall、F1和混淆矩阵的预测值。需要使用cross_val_predict获取所有样本的交叉验证预测结果，才能正确计算这些多维度指标。
• 分类器实例状态残留：循环中复用同一个分类器实例，Hold-Out分支已经完成训练，后续交叉验证分支即使调用cross_val_score重新训练，也可能因实例状态残留影响结果。建议每次循环创建新的分类器实例，避免状态污染。

修正代码示例 #

from sklearn.model_selection import cross_val_predict

# Perform classification and evaluation for each classifier and cross-validation method
for clf_name, clf in classifiers.items():
    print(f"Classifier: {clf_name}")
    # 每次循环创建新的分类器实例，避免状态残留
    clf_instance = clf.__class__(**clf.get_params())
    for cv_name, cv_method in cv_methods.items():
        if cv_name == 'Hold-Out Method':
            X_train_cv, X_test_cv, y_train_cv, y_test_cv = cv_method
            clf_instance.fit(X_train_cv, y_train_cv)
            y_pred = clf_instance.predict(X_test_cv)
        else:
            # 使用cross_val_predict获取所有样本的交叉验证预测值
            y_pred = cross_val_predict(clf_instance, X, y, cv=cv_method)
            y_test_cv = y  # 交叉验证中所有样本都会作为测试集参与一次
        
        # 计算并打印指标
        accuracy = accuracy_score(y_test_cv, y_pred)
        precision = precision_score(y_test_cv, y_pred, average='weighted')
        recall = recall_score(y_test_cv, y_pred, average='weighted')
        f1 = f1_score(y_test_cv, y_pred, average='weighted')
        confusion = confusion_matrix(y_test_cv, y_pred)
        
        print(f"\n{cv_name} Metrics for {clf_name}:")
        print(f"Accuracy: {accuracy:.9f}")
        print(f"Precision: {precision:.4f}")
        print(f"Recall: {recall:.4f}")
        print(f"F1 Score: {f1:.4f}")
        print("Confusion Matrix:")
        print(confusion)

内容的提问来源于stack exchange，提问作者cigsesgi