Pipeline转DataFrame时出现形状不匹配ValueError的解决方法

错误原因 #

报错ValueError: Shape of passed values is (8631, 28), indices imply (8631, 17)的核心问题是：

• 使用OneHotEncoder处理分类特征时，会将每个多分类特征拆分成多个二进制列，导致预处理后的数据列数（28列）远多于原始特征列数（17列）。
• 但你在FunctionTransformer里仍使用原始特征列名列表names构造DataFrame，列数不匹配直接触发报错。

解决方案 #

需要动态生成预处理后所有特征的正确列名，替换原始列名列表。具体步骤如下：

1. 修正分类特征筛选逻辑 #

原代码中分类特征的筛选规则可能有误，改为精准筛选object或category类型的列，避免漏选/多选：

categorical_cols = X.select_dtypes(include=["object", "category"]).columns.tolist()
numerical_cols = X.select_dtypes(exclude=["object", "category"]).columns.tolist()

2. 拟合预处理模块并获取完整列名 #

先单独拟合ColumnTransformer，分别提取数值特征列名、编码后的分类特征列名，以及remainder='passthrough'保留的列名，最后合并成完整列名列表：

preprocessor = ColumnTransformer(
    transformers=[
        ('num', numeric_transformer, numerical_cols),
        ('cat', categorical_transformer, categorical_cols)
    ],
    remainder='passthrough'
)

# 先拟合预处理模块
preprocessor.fit(X_train)

# 获取数值特征列名
num_cols = preprocessor.named_transformers_['num'].get_feature_names_out(numerical_cols)
# 获取OneHot编码后的分类特征列名
cat_cols = preprocessor.named_transformers_['cat'].get_feature_names_out(categorical_cols)
# 获取remainder保留的列名（未被num/cat处理的列）
remainder_cols = preprocessor.transformers_[2][2]

# 合并所有列名
all_cols = list(num_cols) + list(cat_cols) + list(remainder_cols)

3. 修改Pipeline的DataFrame转换逻辑 #

用生成的完整列名列表all_cols替换原始的names，确保列数匹配：

pipe_preprocessor = Pipeline(steps=[
    ("preprocessor", preprocessor),
    ("pandarizer", FunctionTransformer(lambda x: pd.DataFrame(x, columns=all_cols)))
])

# 拟合管道并转换数据
pipe_preprocessor.fit(X_train)
X_train_pipe = pipe_preprocessor.transform(X_train)
X_test_pipe = pipe_preprocessor.transform(X_test)

完整修正后代码 #

from sklearn.preprocessing import FunctionTransformer, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
import pandas as pd
from sklearn.model_selection import train_test_split

print("step1: import lib")
print("step2: loading raw data")
df = pd.read_csv("online_shoppers_intention.csv")

print("step3: data preparition")
X = df.drop(['Revenue'], axis=1)
y = df['Revenue']

print("step4: data splitting")
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.3, random_state=0)

numeric_transformer = SimpleImputer(strategy='constant')
categorical_transformer = OneHotEncoder(handle_unknown='ignore')

# 修正分类/数值特征筛选逻辑
categorical_cols = X.select_dtypes(include=["object", "category"]).columns.tolist()
numerical_cols = X.select_dtypes(exclude=["object", "category"]).columns.tolist()

preprocessor = ColumnTransformer(
    transformers=[
        ('num', numeric_transformer, numerical_cols),
        ('cat', categorical_transformer, categorical_cols)
    ],
    remainder='passthrough'
)

# 拟合预处理模块并生成完整列名
preprocessor.fit(X_train)
num_cols = preprocessor.named_transformers_['num'].get_feature_names_out(numerical_cols)
cat_cols = preprocessor.named_transformers_['cat'].get_feature_names_out(categorical_cols)
remainder_cols = preprocessor.transformers_[2][2]
all_cols = list(num_cols) + list(cat_cols) + list(remainder_cols)

# 构建Pipeline并转换数据
pipe_preprocessor = Pipeline(steps=[
    ("preprocessor", preprocessor),
    ("pandarizer", FunctionTransformer(lambda x: pd.DataFrame(x, columns=all_cols)))
]).fit(X_train)

X_train_pipe = pipe_preprocessor.transform(X_train)
X_test_pipe = pipe_preprocessor.transform(X_test)

内容的提问来源于stack exchange，提问作者Manoj GH