手动分层KFold交叉验证与cross_val_score得分结果不一致问题

我用Python循环实现分层KFold划分训练集，在循环内训练文本分类器并验证，得到的accuracy、F1指标和sklearn的cross_val_score结果差异明显，预期两种方法结果一致。任务为文本分类，采用TF-IDF向量化，两种实现的代码及输出如下：

手动交叉验证代码 #

#Importing metrics functions to measure performance of a  model
from sklearn.metrics import f1_score, accuracy_score, precision_score, recall_score
from sklearn.model_selection import StratifiedKFold
data_validation = []  # list used to store the results of model validation using cross validation
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
accuracy_val = []
f1_val = []

# use ravel function to flatten the multi-dimensional array to a single dimension
for train_index, val_index in (skf.split(X_train, y_train)):
    X_tr, X_val = X_train.ravel()[train_index], X_train.ravel()[val_index] 
    y_tr, y_val  = y_train.ravel()[train_index] , y_train.ravel()[val_index]
    tfidf=TfidfVectorizer()
    X_tr_vec_tfidf = tfidf.fit_transform(X_tr) # vectorize the training folds
    X_val_vec_tfidf = tfidf.transform(X_val) # vectorize the validation fold    
    #instantiate model 
    model= MultinomialNB(alpha=0.5, fit_prior=False) 
    #Training the empty model with our training dataset
    model.fit(X_tr_vec_tfidf, y_tr)  
    predictions_val = model.predict(X_val_vec_tfidf) # make predictions with the validation dataset
    acc_val = accuracy_score(y_val, predictions_val)
    accuracy_val.append(acc_val)
    f_val=f1_score(y_val, predictions_val)
    f1_val.append(f_val)

avg_accuracy_val = np.mean(accuracy_val)
avg_f1_val = np.mean(f1_val)

# temp list to store the metrics 
temp = ['NaiveBayes']
temp.append(avg_accuracy_val)   #validation accuracy score 
temp.append(avg_f1_val)         #validation f1 score
data_validation.append(temp)    
#Create a table ,using dataframe, which contains the metrics for all the trained and tested ML models
result = pd.DataFrame(data_validation, columns = ['Algorithm','Accuracy Score : Validation','F1-Score  : Validation'])
result.reset_index(drop=True, inplace=True)
result      

手动交叉验证输出 #

Algorithm   Accuracy Score : Validation     F1-Score : Validation
0   NaiveBayes  0.77012                      0.733994

cross_val_score实现代码 #

from sklearn.model_selection import cross_val_score
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
scores = ['accuracy', 'f1']
#Text vectorization of training and testing datasets using NLP technique TF-IDF
tfidf=TfidfVectorizer()
X_tr_vec_tfidf = tfidf.fit_transform(X_train)
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
nb=MultinomialNB(alpha=0.5, fit_prior=False) 
for score in ["accuracy", "f1"]:
    print (f'{score}: {cross_val_score(nb,X_tr_vec_tfidf,y_train,cv=skf,scoring=score).mean()} ')

cross_val_score输出 #

accuracy: 0.7341283583255231 
f1: 0.7062017090972422 

两种方法得到的accuracy和F1指标差异明显，使用KNeighborsClassifier时差异更显著。

问题原因分析 #

核心问题是数据泄露与TF-IDF拟合时机错误：

• 手动实现中，每个fold循环内重新初始化TfidfVectorizer，仅在当前fold的训练集上执行fit_transform，验证集用同一vectorizer做transform，这是正确的交叉验证流程——特征提取仅使用训练集数据，无数据泄露。
• cross_val_score实现中，先对整个X_train做tfidf.fit_transform，再传入交叉验证。这相当于提前用所有训练数据（包括后续作为验证集的部分）拟合TF-IDF的词汇表和逆文档频率，属于严重数据泄露，导致模型验证表现被低估。

另外，f1_score的默认平均策略可能存在差异：手动代码中f1_score()默认用average='binary'，而cross_val_score的scoring='f1'在多分类任务中默认用average='macro'，需根据任务类型统一参数，但核心原因仍是TF-IDF拟合时机错误。

修正后的cross_val_score代码 #

用Pipeline封装TF-IDF和模型，确保每个fold的特征提取仅基于当前训练集：

from sklearn.model_selection import cross_val_score
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
# 封装特征提取与模型
pipeline = Pipeline([
    ('tfidf', TfidfVectorizer()),
    ('nb', MultinomialNB(alpha=0.5, fit_prior=False))
])

for score in ["accuracy", "f1"]:
    # 多分类任务需指定对应平均方式，如scoring='f1_macro'
    score_mean = cross_val_score(pipeline, X_train, y_train, cv=skf, scoring=score).mean()
    print(f'{score}: {score_mean} ')

修正后两种方法结果会一致，因为Pipeline避免了数据泄露，和手动实现的交叉验证逻辑完全对齐。

内容的提问来源于stack exchange，提问作者Tony