使用GridSearchCV训练Pipeline模型时触发ValueError维度不匹配错误

问题现象 #

使用GridSearchCV训练包含自定义特征工程的Pipeline时，触发以下维度不匹配错误，但直接训练该Pipeline时完全正常：

raise ValueError(f"Shape of passed values is {passed}, indices imply {implied}")
ValueError: Shape of passed values is (12610, 54), indices imply (12610, 55)

原代码 #

class FeatureEngineering(BaseEstimator, TransformerMixin):
    def __init__(self):
        pass

    def fit(self, X, y=None):
        return self

    def transform(self, x):
        transformed_data = x.toarray()
        feature_names = preprocessor_1.get_feature_names_out()
        df_transformed = pd.DataFrame(x.toarray(), columns=preprocessor_1.get_feature_names_out())

        if transformed_data.shape[1] != len(feature_names):
          raise ValueError(f"Mismatch: Data has {transformed_data.shape[1]} columns but {len(feature_names)} feature names.")
    
        df_transformed['total_room'] = df_transformed['num_dcr__bedrooms'] + df_transformed['num_dcr__bathrooms']
        df_transformed['total_room_add_floors'] = df_transformed['total_room'] + df_transformed['num_dcr__floors']
        df_transformed['bedrooms_multi_area'] = df_transformed['num_dcr__bedrooms'] * df_transformed['num_cont__area']
        df_transformed['bathrooms_multi_area'] = df_transformed['num_dcr__bathrooms'] * df_transformed['num_cont__area']
        df_transformed['area_floors'] = df_transformed['num_cont__area'] * df_transformed['num_dcr__floors']
        print(df_transformed.shape)
        return df_transformed

num_dcr_cols = ['bathrooms', 'bedrooms', 'floors']
num_cont_cols = ['area', 'frontage', 'access_road']
cat_nom_cols = ['legal_status', 'province']
cat_ord_cols = ['furniture_state']
comp_fe = ['bathrooms', 'bedrooms', 'floors', 'area']

cat_ord = ["Full", "Basic", "No furniture"]

# 离散列处理
num_dcr_transformer = Pipeline(steps=[
    ('imputer', dcr_imputer),
    ('scaler', StandardScaler())
])

# 连续列处理
num_cont_transformer = Pipeline(steps=[
    ('imputer', cont_imputer),
    ('scaler', StandardScaler())
])

# 标称类别列处理
cat_nom_transformer = Pipeline(steps=[
    ('encoder', OneHotEncoder(handle_unknown='ignore'))
])

# 有序类别列处理
cat_ord_transformer = Pipeline(steps=[
    ('encoder', OrdinalEncoder(categories=[cat_ord]))
])

# 特征工程Pipeline
fe = Pipeline(steps=[
    ('fe', FeatureEngineering())
])

# 预处理整合
preprocessor_1 = ColumnTransformer(
    transformers=[
        ('cat_nom', cat_nom_transformer, cat_nom_cols),
        ('cat_ord', cat_ord_transformer, cat_ord_cols),
        ('num_dcr', num_dcr_transformer, num_dcr_cols),
        ('num_cont', num_cont_transformer, num_cont_cols),
    ])

preprocessor_2 = Pipeline(steps=[
    ('preprocessor_1', preprocessor_1),
    ('fe', fe)
])

# 完整模型Pipeline
pipeline_rf = Pipeline(
    steps=[
        ('preprocessor_2', preprocessor_2),
        ('rf', RandomForestRegressor(random_state=1))
    ]
)

# 网格搜索参数
params = {
    "rf__n_estimators": [50,100],
    "rf__max_depth": [10,20],
    "rf__min_samples_split": [5, 10]
}

# 启动网格搜索
model_rf_cv = GridSearchCV(pipeline_rf, param_grid=params, cv=3, n_jobs=-1,verbose=4, scoring="r2")
model_rf_cv.fit(x_train, y_train)
y_pred = model_rf_cv.predict(x_test)
print("最优参数: ", model_rf_cv.best_params_)
print("验证集R2得分: ", model_rf_cv.best_score_)
print("测试集R2得分: ", r2_score(y_test, y_pred))

问题根源 #

问题出在自定义FeatureEngineering类依赖全局变量preprocessor_1：

• 直接训练时，preprocessor_1基于整个训练集拟合，特征名数量和转换后的数据列数完全匹配。
• 但GridSearchCV启用并行（n_jobs=-1）时，每个CV折的训练在独立进程中运行：
  1. 每个进程会重新拟合preprocessor_1，如果验证集中的标称类别（比如province）有训练集未出现的取值，OneHotEncoder(handle_unknown='ignore')会忽略这些类别，导致转换后的数据列数减少。
  2. 但FeatureEngineering.transform()中调用的是全局的preprocessor_1.get_feature_names_out()，该特征名是基于整个训练集生成的，列数更多，最终创建DataFrame时出现维度不匹配。

修复方案 #

修改FeatureEngineering类，在fit阶段从传入的预处理数据中获取特征名并保存为实例属性，避免依赖全局变量：

修改后的FeatureEngineering类 #

class FeatureEngineering(BaseEstimator, TransformerMixin):
    def __init__(self):
        self.feature_names = None

    def fit(self, X, y=None):
        # 从当前拟合完成的ColumnTransformer中获取特征名
        self.feature_names = X.get_feature_names_out()
        return self

    def transform(self, X):
        transformed_data = X.toarray()
        # 使用实例属性中保存的特征名，而非全局变量
        df_transformed = pd.DataFrame(transformed_data, columns=self.feature_names)

        if transformed_data.shape[1] != len(self.feature_names):
            raise ValueError(f"Mismatch: Data has {transformed_data.shape[1]} columns but {len(self.feature_names)} feature names.")
        
        df_transformed['total_room'] = df_transformed['num_dcr__bedrooms'] + df_transformed['num_dcr__bathrooms']
        df_transformed['total_room_add_floors'] = df_transformed['total_room'] + df_transformed['num_dcr__floors']
        df_transformed['bedrooms_multi_area'] = df_transformed['num_dcr__bedrooms'] * df_transformed['num_cont__area']
        df_transformed['bathrooms_multi_area'] = df_transformed['num_dcr__bathrooms'] * df_transformed['num_cont__area']
        df_transformed['area_floors'] = df_transformed['num_cont__area'] * df_transformed['num_dcr__floors']
        return df_transformed

调整Pipeline结构 #

无需单独定义fe Pipeline，直接在preprocessor_2中使用修改后的FeatureEngineering：

preprocessor_2 = Pipeline(steps=[
    ('preprocessor_1', preprocessor_1),
    ('fe', FeatureEngineering())
])

修复原理 #

• 每个CV折的进程中，preprocessor_1拟合完成后，FeatureEngineering.fit()会获取当前进程中拟合后的特征名并保存。
• transform()阶段使用该进程内的特征名，确保和转换后的数据列数完全匹配，彻底避免维度不匹配问题。

内容的提问来源于stack exchange，提问作者Đỗ Nhật Nam