为何sklearn cross_val_score对不同阶多项式返回相同MSE？

问题现象 #

使用sklearn进行多项式回归交叉验证时，预期不同阶数的模型会输出不同的MSE，但运行代码后所有阶数返回的MSE完全相同。

错误代码 #

import numpy as np
import pandas as pd
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score, train_test_split

np.random.seed(1)

url = 'https://raw.githubusercontent.com/selva86/datasets/master/Auto.csv'
data = pd.read_csv(url, na_values='?').dropna()

X,y = data['horsepower'], data.mpg

X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.5)

lm = LinearRegression()

for i in range(1,6):
    poly = PolynomialFeatures(degree=i)
    X_train_poly = poly.fit_transform(X_train.values.reshape(-1,1))
    model = lm.fit(X_train_poly, y_train)
    scores_poly = cross_val_score(model, X_test.values.reshape(-1,1),y_test,scoring='neg_mean_squared_error',
                            cv = len(X_test), n_jobs = -1)
    
    print(f'Degree-{i} polynomial, MSE: {abs(scores_poly).mean()}')

错误输出 #

Degree-1 polynomial, MSE: 25.214173196098535
Degree-2 polynomial, MSE: 25.214173196098535
Degree-3 polynomial, MSE: 25.214173196098535
Degree-4 polynomial, MSE: 25.214173196098535
Degree-5 polynomial, MSE: 25.214173196098535

修正后的实现 #

通过对测试集执行对应的多项式特征转换，得到了符合预期的不同MSE结果。

修正后代码 #

import numpy as np
import pandas as pd
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score, train_test_split

np.random.seed(1)

url = 'https://raw.githubusercontent.com/selva86/datasets/master/Auto.csv'
data = pd.read_csv(url, na_values='?').dropna()

X,y = data['horsepower'].values.reshape(-1,1), data.mpg.values.reshape(-1,1)

X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.5)

lm = LinearRegression()

for i in range(1,6):
    poly = PolynomialFeatures(degree=i)
    X_train_poly = poly.fit_transform(X_train)
    X_test_poly = poly.transform(X_test)  # 更规范的写法：用训练集拟合的转换器转换测试集
    model = lm.fit(X_train_poly, y_train)
    scores_poly = cross_val_score(model, X_test_poly,y_test,scoring='neg_mean_squared_error',
                            cv = len(X_test), n_jobs = -1)
    
    print(f'Degree-{i} polynomial, MSE: {abs(scores_poly).mean()}')

修正后输出 #

Degree-1 polynomial, MSE: 25.214173196098535
Degree-2 polynomial, MSE: 18.678068281482577
Degree-3 polynomial, MSE: 18.85993623145088
Degree-4 polynomial, MSE: 19.085567664927655
Degree-5 polynomial, MSE: 18.79973878463155

原因解析 #

1. 特征维度不匹配
   原代码中，模型是基于i阶多项式转换后的训练特征（维度为i+1，包含常数项、x、x²...x^i）训练的，但交叉验证时传入的是原始1维测试特征。sklearn为了匹配模型输入维度，会自动将低维特征补0填充到模型期望的维度，这相当于所有高阶项的输入都是0，模型无法利用训练得到的高阶系数进行预测，退化为线性回归，因此所有阶数的MSE完全一致。

2. 测试集特征转换缺失
   修正后的代码对测试集执行了与训练集一致的多项式特征转换，让测试集的特征维度和模型训练时的输入维度匹配。此时模型可以正常调用所有阶数的系数进行预测，不同阶数的模型拟合能力差异会体现在MSE上，得到符合预期的结果。

注意：更规范的做法是用poly.transform(X_test)而非fit_transform，因为fit步骤已经在训练集完成，测试集只需复用训练集的特征转换规则，避免因重新拟合导致的数据分布不一致问题。

内容的提问来源于stack exchange，提问作者guin0x