删除含缺失值列后sklearn仍抛出ValueError问题求助

我正在学习Kaggle中级机器学习课程，刚接触机器学习。目前尝试构建Random Forest模型并对数据进行One-Hot Encoding（OH编码），首次操作遇到了问题。

为简化处理，我删除了所有含缺失值的列：

import pandas as pd
from sklearn.model_selection import train_test_split

X = pd.read_csv('/kaggle/input/home-data-for-ml-course/train.csv', index_col='Id') 
X_test = pd.read_csv('/kaggle/input/home-data-for-ml-course/test.csv', index_col='Id')

X.dropna(axis=0, subset=['SalePrice'], inplace=True)
y = X.SalePrice
X.drop(['SalePrice'], axis=1, inplace=True)

cols_with_missing = [col for col in X.columns if X[col].isnull().any()] 
X.drop(cols_with_missing, axis=1, inplace=True)
X_test.drop(cols_with_missing, axis=1, inplace=True)

X_train, X_valid, y_train, y_valid = train_test_split(X, y,
                                                      train_size=0.8, test_size=0.2,
                                                      random_state=0)

随后对数据进行One-Hot Encoding：

from sklearn.preprocessing import OneHotEncoder

object_cols = [col for col in X_train.columns if X_train[col].dtype == "object"]

low_cardinality_cols = [col for col in object_cols if X_train[col].nunique() < 10]

num_X_train = X_train.drop(object_cols, axis=1)
num_X_valid = X_valid.drop(object_cols, axis=1)
num_X_test = X_test.drop(object_cols, axis=1)

OH_encoder = OneHotEncoder(handle_unknown='ignore', sparse=False)

OH_cols_train = pd.DataFrame(OH_encoder.fit_transform(X_train[low_cardinality_cols]))
OH_cols_valid = pd.DataFrame(OH_encoder.transform(X_valid[low_cardinality_cols]))
OH_cols_test = pd.DataFrame(OH_encoder.transform(X_test[low_cardinality_cols]))

OH_cols_train.index = X_train.index
OH_cols_valid.index = X_valid.index
OH_cols_test.index = X_test.index

OH_X_train = pd.concat([num_X_train, OH_cols_train], axis=1)
OH_X_valid = pd.concat([num_X_valid, OH_cols_valid], axis=1)
OH_X_test = pd.concat([num_X_test, OH_cols_test], axis=1)

创建模型并预测时：

from sklearn.ensemble import RandomForestRegressor

model = RandomForestRegressor(n_estimators=100, random_state=0)
model.fit(OH_X_train, y_train)
preds = model.predict(OH_X_test)

触发错误：

ValueError: Input contains NaN, infinity or a value too large for dtype('float32').

训练集和测试集处理方式一致，模型训练正常但预测出错，恳请帮助解决。

错误原因 #

你删除的是训练集中存在缺失值的列，但测试集的保留列（尤其是数值列）可能仍有缺失值。训练集处理后无缺失，所以模型能正常训练，但测试集的数值列存在NaN，导致拼接后的OH_X_test包含缺失值，触发预测错误。

解决步骤 #

1. 检查测试集数值列的缺失情况
2. 对数值列进行缺失值填充（树模型推荐用中位数，避免均值受异常值影响）
3. 确保训练/验证/测试集的数值列处理逻辑一致

修改后的代码 #

在处理数值列时加入缺失值填充逻辑：

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder
from sklearn.ensemble import RandomForestRegressor

# 读取数据
X = pd.read_csv('/kaggle/input/home-data-for-ml-course/train.csv', index_col='Id') 
X_test = pd.read_csv('/kaggle/input/home-data-for-ml-course/test.csv', index_col='Id')

# 处理目标变量
X.dropna(axis=0, subset=['SalePrice'], inplace=True)
y = X.SalePrice
X.drop(['SalePrice'], axis=1, inplace=True)

# 删除训练集中有缺失值的列
cols_with_missing = [col for col in X.columns if X[col].isnull().any()] 
X.drop(cols_with_missing, axis=1, inplace=True)
X_test.drop(cols_with_missing, axis=1, inplace=True)

# 划分训练/验证集
X_train, X_valid, y_train, y_valid = train_test_split(X, y,
                                                      train_size=0.8, test_size=0.2,
                                                      random_state=0)

# 区分对象列和数值列
object_cols = [col for col in X_train.columns if X_train[col].dtype == "object"]
num_cols = [col for col in X_train.columns if X_train[col].dtype in ['int64', 'float64']]

# 筛选低基数分类列
low_cardinality_cols = [col for col in object_cols if X_train[col].nunique() < 10]

# 处理数值列：用训练集的中位数填充所有数据集的缺失值
num_median = X_train[num_cols].median()
num_X_train = X_train[num_cols].fillna(num_median)
num_X_valid = X_valid[num_cols].fillna(num_median)
num_X_test = X_test[num_cols].fillna(num_median)

# One-Hot编码分类列
OH_encoder = OneHotEncoder(handle_unknown='ignore', sparse=False)
OH_cols_train = pd.DataFrame(OH_encoder.fit_transform(X_train[low_cardinality_cols]))
OH_cols_valid = pd.DataFrame(OH_encoder.transform(X_valid[low_cardinality_cols]))
OH_cols_test = pd.DataFrame(OH_encoder.transform(X_test[low_cardinality_cols]))

# 恢复索引
OH_cols_train.index = X_train.index
OH_cols_valid.index = X_valid.index
OH_cols_test.index = X_test.index

# 拼接数值列和编码后的分类列
OH_X_train = pd.concat([num_X_train, OH_cols_train], axis=1)
OH_X_valid = pd.concat([num_X_valid, OH_cols_valid], axis=1)
OH_X_test = pd.concat([num_X_test, OH_cols_test], axis=1)

# 训练模型并预测
model = RandomForestRegressor(n_estimators=100, random_state=0)
model.fit(OH_X_train, y_train)
preds = model.predict(OH_X_test)

额外检查 #

可以在预测前验证OH_X_test是否还有缺失值：

print(OH_X_test.isnull().sum().sum())

如果输出为0，说明数据已无缺失，可正常预测。

内容的提问来源于stack exchange，提问作者tyl3366