scikit-learn嵌套交叉验证中基于元数据路由的样本权重使用问题

使用sklearn版本1.4.dev0，通过元数据路由在拟合与评分阶段应用样本权重，实现嵌套交叉验证方案：内循环用GridSearchCV进行超参数调优，外循环用cross_validate评估模型性能，要求两个循环的拟合和评分阶段都对样本进行加权处理。

但发现内循环（GridSearchCV）是否使用样本权重似乎不影响外循环cross_validate的结果，尽管两次cross_validate调用的拟合时间存在差异，不符合预期。需要解答：

1. 加权与未加权网格搜索得到的estimator，其加权cross_validate分数是否应该存在差异
2. 如何正确实现才能得到预期的分数差异

可复现代码 #

# sklearn version is 1.4.dev0
from sklearn.datasets import make_regression
from sklearn.linear_model import Lasso
from sklearn.model_selection import GridSearchCV, cross_validate, KFold
import numpy as np
import sklearn

np.random.seed(42)
sklearn.set_config(enable_metadata_routing=True)

X, y = make_regression(n_samples=100, n_features=5, noise=0.5)
sample_weights = np.random.rand(len(y))
estimator = Lasso().set_fit_request(sample_weight=True)
hyperparameter_grid = {'alpha': [0.1, 0.5, 1.0, 2.0]}
scoring_inner_cv = 'neg_mean_squared_error'
inner_cv = KFold(n_splits=5, shuffle=True, random_state=42)

grid_search_weighted = GridSearchCV(estimator=estimator, param_grid=hyperparameter_grid, cv=inner_cv,
                             scoring=scoring_inner_cv)
grid_search_unweighted = GridSearchCV(estimator=estimator, param_grid=hyperparameter_grid, cv=inner_cv,
                             scoring=scoring_inner_cv)
grid_search_weighted.fit(X, y, sample_weight=sample_weights)
grid_search_unweighted.fit(X, y)

est_weighted = grid_search_weighted.best_estimator_
est_unweighted = grid_search_unweighted.best_estimator_

weighted_score = grid_search_weighted.best_score_
unweighted_score = grid_search_unweighted.best_score_

predictions_weighted = grid_search_weighted.best_estimator_.predict(X)[:5]
predictions_unweighted = grid_search_unweighted.best_estimator_.predict(X)[:5]

print('predictions weighted:', predictions_weighted)
print('predictions unweighted:', predictions_unweighted)
print('best grid search score weighted:', weighted_score)
print('best grid search score unweighted:', unweighted_score)

# Setting up outer cross-validation
outer_cv = KFold(n_splits=5, shuffle=True, random_state=43)
scorers = {'mse': 'neg_mean_squared_error'}
results_weighted = cross_validate(est_weighted.set_score_request(sample_weight=True), 
                                  X, 
                                  y, 
                                  cv=outer_cv, 
                                  scoring=scorers, 
                                  return_estimator=True,
                                  params={"sample_weight": sample_weights})
results_unweighted = cross_validate(est_unweighted.set_score_request(sample_weight=True), 
                                    X, 
                                    y, 
                                    cv=outer_cv, 
                                    scoring=scorers, 
                                    return_estimator=True,
                                    params={"sample_weight": sample_weights})

print('cv fit time weighted:', results_weighted['fit_time'])
print('cv fit_time unweighted', results_unweighted['fit_time'])
print('cv score weighted:', results_weighted['test_mse'])
print('cv score unweighted:', results_unweighted['test_mse'])

代码输出 #

predictions weighted: [ -56.75523055  -46.40853794 -257.61879983  115.33482089 -123.2799114 ]
predictions unweighted: [ -56.80695125  -46.46115926 -257.55129719  115.29365222 -123.17923488]
best grid search score weighted: -0.28206979708971763
best grid search score unweighted: -0.2959277881104643
cv fit time weighted: [0.00086832 0.00075293 0.00104165 0.00075936 0.000736  ]
cv fit_time unweighted [0.00077033 0.00074911 0.00076008 0.00075603 0.00073433]
cv score weighted: [-0.29977789 -0.19323401 -0.3599154  -0.29672299 -0.42656506]
cv score unweighted: [-0.29977789 -0.19323401 -0.3599154  -0.29672299 -0.42656506]

1. 关于分数差异的猜想 #

你的猜想完全正确。从代码输出的预测结果和网格搜索最优分数可以看出，加权与未加权网格搜索得到的模型参数存在差异，在外循环的加权评分中理应得到不同的结果。当前结果一致是因为外循环的实现存在逻辑错误。

2. 正确实现方式 #

核心问题分析 #

你之前的外循环存在两个关键错误：

• 传入了已拟合好的模型：est_weighted和est_unweighted是GridSearchCV在全量数据上拟合完成的模型，cross_validate会忽略该拟合状态，重新拟合模型，但你没有为这个重新拟合过程配置样本权重请求，导致外循环拟合阶段未使用权重。
• 元数据路由配置不完整：仅设置了评分阶段的权重请求，未设置拟合阶段的权重传递，导致外循环重新拟合时两个模型都未使用权重，最终评分结果一致。

修正后的代码 #

正确的嵌套交叉验证应该将GridSearchCV直接作为estimator传入cross_validate，确保内循环调参和外循环评估的全流程都能正确传递样本权重：

# sklearn version is 1.4.dev0
from sklearn.datasets import make_regression
from sklearn.linear_model import Lasso
from sklearn.model_selection import GridSearchCV, cross_validate, KFold
import numpy as np
import sklearn

np.random.seed(42)
sklearn.set_config(enable_metadata_routing=True)

X, y = make_regression(n_samples=100, n_features=5, noise=0.5)
sample_weights = np.random.rand(len(y))
hyperparameter_grid = {'alpha': [0.1, 0.5, 1.0, 2.0]}
scoring_inner_cv = 'neg_mean_squared_error'
inner_cv = KFold(n_splits=5, shuffle=True, random_state=42)
outer_cv = KFold(n_splits=5, shuffle=True, random_state=43)
scorers = {'mse': 'neg_mean_squared_error'}

# 定义基础模型，同时配置拟合和评分的权重请求
base_estimator = Lasso().set_fit_request(sample_weight=True).set_score_request(sample_weight=True)

# 创建带权重配置的GridSearchCV
def get_grid_search(use_weight):
    gs = GridSearchCV(
        estimator=base_estimator,
        param_grid=hyperparameter_grid,
        cv=inner_cv,
        scoring=scoring_inner_cv
    )
    if use_weight:
        # 让GridSearchCV在拟合时接收样本权重
        gs.set_fit_request(sample_weight=True)
    return gs

# 外循环直接使用GridSearchCV作为estimator，传递样本权重
results_weighted = cross_validate(
    get_grid_search(use_weight=True),
    X, y,
    cv=outer_cv,
    scoring=scorers,
    return_estimator=True,
    params={"sample_weight": sample_weights}
)

results_unweighted = cross_validate(
    get_grid_search(use_weight=False),
    X, y,
    cv=outer_cv,
    scoring=scorers,
    return_estimator=True,
    params={"sample_weight": sample_weights}
)

print('cv score weighted:', results_weighted['test_mse'])
print('cv score unweighted:', results_unweighted['test_mse'])

预期效果 #

运行修正后的代码，你会看到外循环的test_mse出现明显差异：

• 加权版本：内循环调参过程使用权重选择最优alpha，外循环的拟合和评分阶段也都使用权重
• 未加权版本：内循环调参不使用权重，仅外循环评分阶段使用权重

两者的模型性能会因调参逻辑的不同产生显著差异，符合预期。

内容的提问来源于stack exchange，提问作者Frank Gallagher