Poisson回归预测工具价格时遇ColumnTransformer特征数不匹配错误

问题背景 #

使用Poisson回归模型进行工具价格预测时，训练流程正常，但测试数据预测阶段触发错误：X has 7 features, but ColumnTransformer expects 13 features，测试数据列数看似与训练时一致，无法定位问题。

训练流程代码 #

# --- Load and Prepare Data ---
y = train['PriceToday']
X = train.drop(columns=['PriceToday'])

# Define non-standard types
non_standard_types = ["Nar", "Orch", "Fru","Comp"]

# Create a flag feature for non-standard
X["Non_Standard_Flag"] = X["Type_LS"].isin(non_standard_types).astype(int)

# Identify numerical and categorical columns
num_features = ["AGE", "POWER", "Hours", "Non_Standard_Flag"]
cat_features = ["BRAND", "Country", "Final_Trans"]

# Define preprocessing pipeline
preprocessor = ColumnTransformer(
    transformers=[
        ('num', StandardScaler(), num_features),
        ('cat', OneHotEncoder(handle_unknown='ignore'), cat_features)
    ], remainder="drop"
)

# --- Train/Test Split ---
# Create a weight column
train["sample_weight"] = train["Type_LS"].apply(lambda x: 1 if x == "Standard" else 5)

train["stratify_group"] = train["BRAND"].astype(str)
X_train, X_val, y_train, y_val, train_weights, val_weights = train_test_split(
    X, y, train["sample_weight"], test_size=0.2, random_state=42, stratify=train["stratify_group"]
)
# Fit the preprocessor once on training data
X_train_preprocessed = preprocessor.fit_transform(X_train)
X_val_preprocessed = preprocessor.transform(X_val)

# Define models
models = {
    "Poisson": PoissonRegressor(alpha=0.01)
}

# Train and evaluate models
model_results = {}

for model_name, model in models.items():
    model.fit(X_train_preprocessed, y_train, sample_weight=train_weights)

    # Predictions
    predictions = model.predict(X_val_preprocessed)
    # Calculate Metrics
    r2 = r2_score(y_val, predictions)

    model_results[model_name] = {
        "model": model,
        "R2": r2
    }

测试数据及处理代码 #

测试数据构造 #

# Ensure new data has the correct format
new_data = pd.DataFrame({
    "AGE": [12, 24, 36, 48, 60, 72, 84, 12, 24, 36, 48, 60, 72, 84],
    "Hours": [500, 1000, 1500, 2000, 2500, 3000, 3500, 500, 1000, 1500, 2000, 2500, 3000, 3500],
    "BRAND": ["NH"] * 14,
    "POWER": [150] * 7 + [80] * 7,
    "Final_Trans": ["Cv"] * 14,
    "Country": ["DEU"] * 14,
    "Type_LS": ["Nar", "Nar", "Nar", "ST", "ST", "ST", "ST", "ST","ST", "ST", "ST", "ST", "ST", "ST"],
    "Current_Pred": [105614, 96681, 88504, 81018, 74165, 67892, 62150, 42608, 39728, 37043, 34540, 32206, 30029, 28000]
})

测试数据处理与预测 #

new_df = pd.DataFrame(new_data)
# Create the 'Non_Standard_Flag'
new_df["Non_Standard_Flag"] = new_df["Type_LS"].isin(non_standard_types).astype(int)

# Select the columns required by the preprocessor
X_new = new_df[['AGE', 'POWER', 'Hours', 'Non_Standard_Flag', 'BRAND', 'Country', 'Final_Trans']]

X_new_preprocessed = preprocessor.transform(X_new) 

# Get the column names after one-hot encoding from the training data
ohe = preprocessor.named_transformers_['cat']
encoded_cat_columns = ohe.get_feature_names_out(cat_features)

# Create column names for numeric features
num_columns = num_features

# Combine the column names
all_columns = num_columns + list(encoded_cat_columns)

# Create a DataFrame from the preprocessed data
X_new_preprocessed_df = pd.DataFrame(X_new_preprocessed, columns=all_columns)

# --- Predict with the Poisson Model ---
poisson_model = model_results["Poisson"]["model"]  # Access the trained Poisson model
predicted_prices = poisson_model.predict(X_new_preprocessed_df)

# Compare and Store Results ---
new_df['Predicted_Price'] = predicted_prices

# Calculate the difference between predicted and current prices
new_df['Price_Difference'] = new_df['Predicted_Price'] - new_df['Current_Pred']

错误信息 #

X has 7 features, but ColumnTransformer expects 13 features

问题原因与解决方案 #

核心原因 #

训练阶段的X是train.drop(columns=['PriceToday'])，意味着它包含train中除PriceToday外的全部13列；而测试阶段的X_new仅筛选了7列。ColumnTransformer在fit时会记录训练数据的输入总列数，预测时会先校验输入列数是否匹配，即使未选中的列会被丢弃。

解决方案 #

1. 补全测试数据的原始列
   确保测试数据包含训练数据train中除PriceToday外的所有原始列，即使这些列不会被预处理流程使用。

2. 训练阶段提前筛选特征列
   修改训练代码，直接筛选出需要的列，避免ColumnTransformer感知到多余列：

   # 训练阶段明确筛选所需列
   required_cols = ['AGE', 'POWER', 'Hours', 'BRAND', 'Country', 'Final_Trans', 'Type_LS']
   X = train[required_cols].copy()
   # 生成Non_Standard_Flag
   X["Non_Standard_Flag"] = X["Type_LS"].isin(non_standard_types).astype(int)
   # 移除Type_LS，因为它不是最终特征
   X = X.drop(columns=['Type_LS'])
   

   此时训练数据的X列数与测试阶段的X_new一致（7列），ColumnTransformer不会再触发列数不匹配错误。

3. 修正测试数据语法错误
   测试数据构造代码中Type_LS行末尾缺少逗号，需补充以避免语法报错。

内容的提问来源于stack exchange，提问作者H_H