IterativeImputer+Bayesian Ridge单特征插补遇样本数不匹配ValueError

问题 #

我编写了imputer_regressor_bay_ridge()函数，尝试用IterativeImputer结合BayesianRidge对训练数据做插补，但始终无法满足IterativeImputer要求的2D数组格式。我的训练数据是包含250个特征的DataFrame，但仅需对单个特征进行插补。

原本计划通过data[feature].values提取单特征的numpy数组，再用.reshape(-1,1)转换为2D数组，但流程存在问题，现在报错样本数不匹配。同时想了解：针对单特征列，有没有比RepeatedStratifiedKFold更合适的交叉验证方法？

训练数据形状输出 #

print(train_data.shape)
data_array = train_data.values
data_array = data_array.reshape(-1, 1)
print(data_array.shape)
data_array

# 输出结果
(1460, 250)
(365000, 1)
array([[-1.73086488],
       [-0.20803433],
       [-0.20714171],
       ...,
       [-0.11785113],
       [ 0.4676514 ],
       [-0.30599503]])

现有代码 #

def imputer_regressor_bay_ridge(data, y):
    data_array = data.values
    data = data_array.reshape(-1, 1)
    interative_imputer = IterativeImputer(BayesianRidge())
    interative_imputer_fit = interative_imputer.fit(data)
    data_imputed = interative_imputer_fit.transform(data)
    cv = linear_model.LinearRegression()
    scores = cross_val_score(interative_imputer, data, 
                             y, scoring='accuracy', cv=cv, n_jobs=-1, error_score='raise')
    
    return scores, data_imputed

调用代码：

scores, data_imputed = imputer_bay_ridge(train_data)

print('Impute Bay Ridge Mean Accuracy: %.3f (%.3f)' % (mean(scores), std(scores)))

错误信息 #

ValueError                                Traceback (most recent call last)
Cell In[4], line 169
    167 #train_data, test_data = minmaxscaler(train_data, test_data)  # alternate run for min-max scaler
    168 columns, imputed_df = imputer_regressor(train_data)
--> 169 scores, data_imputed = imputer_regressor_bay_ridge(train_data, test_data)
    171 misTrain = whichColumnsMissing(train_data)
    172 misTest = whichColumnsMissing(test_data)

Cell In[4], line 110, in imputer_regressor_bay_ridge(data, y)
    108 data_imputed = interative_imputer_fit.transform(data)
    109 cv = linear_model.LinearRegression()
--> 110 scores = cross_val_score(interative_imputer, data, 
    111                          y, scoring='accuracy', cv=cv, n_jobs=-1, error_score='raise')
    113 return scores, data_imputed

File ~/opt/anaconda3/lib/python3.9/site-packages/sklearn/model_selection/_validation.py:509, in cross_val_score(estimator, X, y, groups, scoring, cv, n_jobs, verbose, fit_params, pre_dispatch, error_score)
    506 # To ensure multimetric format is not supported
    507 scorer = check_scoring(estimator, scoring=scoring)
--> 509 cv_results = cross_validate(
    510     estimator=estimator,
    511     X=X,
    512     y=y,
    513     groups=groups,
    514     scoring={"score": scorer},
    515     cv=cv,
    516     n_jobs=n_jobs,
    517     verbose=verbose,
    518     fit_params=fit_params,
    519     pre_dispatch=pre_dispatch,
    520     error_score=error_score,
    521 )
    522 return cv_results["test_score"]

File ~/opt/anaconda3/lib/python3.9/site-packages/sklearn/model_selection/_validation.py:253, in cross_validate(estimator, X, y, groups, scoring, cv, n_jobs, verbose, fit_params, pre_dispatch, return_train_score, return_estimator, error_score)
     48 def cross_validate(
     49     estimator,
     50     X,
   (...)
     62     error_score=np.nan,
     63 ):
     64     """Evaluate metric(s) by cross-validation and also record fit/score times.
     65 
     66     Read more in the :ref:`User Guide <multimetric_cross_validation>`.
   (...)
    251 
    252     """
--> 253     X, y, groups = indexable(X, y, groups)
    255     cv = check_cv(cv, y, classifier=is_classifier(estimator))
    257     if callable(scoring):

File ~/opt/anaconda3/lib/python3.9/site-packages/sklearn/utils/validation.py:378, in indexable(*iterables)
    359 """Make arrays indexable for cross-validation.
    360 
    361 Checks consistent length, passes through None, and ensures that everything
   (...)
    374     sparse matrix, or dataframe) or `None`.
    375 """
    377 result = [_make_indexable(X) for X in iterables]
--> 378 check_consistent_length(*result)
    379 return result

File ~/opt/anaconda3/lib/python3.9/site-packages/sklearn/utils/validation.py:332, in check_consistent_length(*arrays)
    330 uniques = np.unique(lengths)
    331 if len(uniques) > 1:
--> 332     raise ValueError(
    333         "Found input variables with inconsistent numbers of samples: %r"
    334         % [int(l) for l in lengths]
    335     )

ValueError: Found input variables with inconsistent numbers of samples: [365000, 1459]

问题分析与解决 #

1. 核心错误原因 #

你现在的代码把整个DataFrame的所有特征摊平成1D数组再转2D，导致样本数从1460变成了1460*250=365000，和传入的y（样本数1459）完全不匹配，这就是报错的根源。

2. 单特征插补的正确姿势 #

如果仅需对单个特征插补，正确做法是先提取该特征列，再转为2D格式：

• 方法1（numpy数组）：

# 假设要插补的特征名为'target_feature'
single_feature = data['target_feature'].values.reshape(-1, 1)

• 方法2（pandas DataFrame）：

# 双层方括号返回DataFrame（天然2D格式），避免Series转数组时的维度问题
single_feature = data[['target_feature']]

修改后的完整函数：

from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer
from sklearn.linear_model import BayesianRidge, LinearRegression
from sklearn.model_selection import KFold, cross_val_score
from sklearn.pipeline import Pipeline
import numpy as np

def imputer_regressor_bay_ridge(data, feature_name, y):
    # 提取单特征的2D格式数据
    single_feature = data[['target_feature']]
    # 用Pipeline整合插补与回归模型（避免交叉验证时的数据泄露）
    pipe = Pipeline([
        ('imputer', IterativeImputer(BayesianRidge())),
        ('regressor', LinearRegression())
    ])
    # 回归任务用KFold交叉验证（分层验证是分类任务用的）
    cv = KFold(n_splits=5, shuffle=True, random_state=42)
    # 回归任务用回归类评分指标，比如负均方误差（转换后为正）
    scores = cross_val_score(pipe, single_feature, y, 
                             scoring='neg_mean_squared_error', cv=cv, n_jobs=-1, error_score='raise')
    # 单独执行插补得到结果
    imputer = IterativeImputer(BayesianRidge())
    data_imputed = imputer.fit_transform(single_feature)
    
    return scores, data_imputed

调用示例：

scores, data_imputed = imputer_regressor_bay_ridge(train_data, 'target_feature', test_data)
# 转换为正的均方误差
mse_scores = -scores
print('Impute Bay Ridge Mean MSE: %.3f (%.3f)' % (mse_scores.mean(), mse_scores.std()))

3. 单特征的交叉验证选择 #

针对单特征的回归任务，推荐以下交叉验证方法：

• KFold/RepeatedKFold：通用的随机划分方式，适合无时间顺序的常规回归数据
• TimeSeriesSplit：如果数据是时间序列，避免未来数据泄露
• ShuffleSplit：适合需要多次随机验证的场景

RepeatedStratifiedKFold是为分类任务设计的（用来保持类别比例），完全不适用于单特征回归任务，不要使用。

内容的提问来源于stack exchange，提问作者Data Science Analytics Manager