OneHotEncoder未转换前序转换器生成的新列,如何解决?
问题原因
ColumnTransformer的各个转换器是并行独立处理原始输入数据的:
date转换器处理原始的visitStartTime列,生成day_of_week、hour等新特征ohe转换器仅处理原始数据中dtype_include='object'的列,不会处理date转换器生成的新分类列
两者结果直接拼接,导致新生成的day_of_week等分类列未被编码。
修正方案
以下两种方式均可解决问题:
方案1:先统一生成时间特征,再全局处理所有分类列
先通过时间转换生成全部衍生特征,再用ColumnTransformer对所有分类特征(包括新生成的)编码,数值特征直接保留。
import pandas as pd from sklearn.pipeline import Pipeline from sklearn.compose import ColumnTransformer from sklearn.preprocessing import OneHotEncoder from sklearn.compose import make_column_selector as selector time_col = ['visitStartTime'] class TimeTransformer: def fit(self, X, y=None): return self def transform(self, X): # 复制输入避免修改原始数据 X_copy = X.copy() for column in X_copy.columns: X_copy['time'] = pd.to_datetime(X_copy[column], unit='s', origin='unix') X_copy['day_of_week'] = X_copy['time'].dt.strftime('%A') X_copy['hour'] = X_copy['time'].dt.hour X_copy['day'] = X_copy['time'].dt.day X_copy['month'] = X_copy['time'].dt.month X_copy['year'] = X_copy['time'].dt.year X_copy = X_copy.drop(['time', column], axis=1) return X_copy # 第一步:处理时间列生成衍生特征 time_transformer = TimeTransformer() # 分类特征编码规则(覆盖新生成的day_of_week和原始object列) ohe_transformer = Pipeline(steps=[ ('ohe', OneHotEncoder(sparse_output=False)) ]) # 数值特征直接保留(包括hour、day等时间衍生列和原始数值列) num_transformer = Pipeline(steps=[]) # 构建完整预处理管道 preprocessor = Pipeline(steps=[ ('time_process', time_transformer), ('feature_process', ColumnTransformer(transformers=[ ('cat', ohe_transformer, selector(dtype_include='object')), ('num', num_transformer, selector(dtype_exclude='object')) ], remainder='passthrough')) ]) # 拟合转换数据 j = preprocessor.fit_transform(X_train)
方案2:在时间处理子管道内单独编码分类特征
针对时间列构建子管道,先生成特征,再对其中的分类列编码、数值列保留;同时处理原始分类列的编码,最后合并所有结果。
import pandas as pd from sklearn.pipeline import Pipeline from sklearn.compose import ColumnTransformer from sklearn.preprocessing import OneHotEncoder from sklearn.compose import make_column_selector as selector time_col = ['visitStartTime'] class TimeTransformer: def fit(self, X, y=None): return self def transform(self, X): X_copy = X.copy() for column in X_copy.columns: X_copy['time'] = pd.to_datetime(X_copy[column], unit='s', origin='unix') X_copy['day_of_week'] = X_copy['time'].dt.strftime('%A') X_copy['hour'] = X_copy['time'].dt.hour X_copy['day'] = X_copy['time'].dt.day X_copy['month'] = X_copy['time'].dt.month X_copy['year'] = X_copy['time'].dt.year X_copy = X_copy.drop(['time', column], axis=1) return X_copy # 时间特征子管道:生成特征后,单独编码分类列、保留数值列 time_feature_process = ColumnTransformer(transformers=[ ('time_cat', OneHotEncoder(sparse_output=False), ['day_of_week']), ('time_num', 'passthrough', ['hour', 'day', 'month', 'year']) ]) time_transformer_pipeline = Pipeline(steps=[ ('time', TimeTransformer()), ('time_feature_encode', time_feature_process) ]) # 原始分类列编码管道 ohe_transformer = Pipeline(steps=[ ('ohe', OneHotEncoder(sparse_output=False)) ]) # 合并所有处理管道 preprocessor = ColumnTransformer(transformers=[ ('date', time_transformer_pipeline, time_col), ('ohe', ohe_transformer, selector(dtype_include='object')) ], remainder='passthrough') j = preprocessor.fit_transform(X_train)
额外优化提示
- 在
TimeTransformer的transform方法中复制输入数据,避免修改原始数据集。 - sklearn 1.2+版本用
OneHotEncoder(sparse_output=False),旧版本替换为sparse=False,方便查看输出结果。
内容的提问来源于stack exchange,提问作者gkl kmr
相关产品推荐
相关产品推荐

