Python中OneHotEncoding、ColumnTransformer与LinearRegression报错排查

在Python脚本中使用ColumnTransformer转换数据集中的分类变量，用于LinearRegression模型。已用OneHotEncoder处理分类变量，转换器输出看似正常，但拟合模型时出现错误：

ValueError: could not convert string to float: 'New Hampshire'

怀疑ColumnTransformer未正确将分类变量转为数值格式，求解决建议。

相关代码如下：

import numpy as np
import pandas as pd 
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression   
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
  
data_set = pd.read_csv('Unemployment in America Per US State.csv')  
  
X = data_set.iloc[:, :-1]
y = data_set.iloc[:, -1]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

ct = ColumnTransformer([('one_hot_encoder', OneHotEncoder(sparse=False), [1])], remainder='passthrough')
X_train = ct.fit_transform(X_train)
X_test = ct.transform(X_test)

regressor = LinearRegression()  
regressor.fit(X_train, y_train)
 
y_pred = regressor.predict(X_test)

• 确认分类列索引是否正确
  错误提示中的New Hampshire是州名，说明你指定的分类列索引[1]大概率不对。先执行print(X.columns)查看特征列顺序，找到州名所在列的真实索引，替换到ColumnTransformer的列参数中。

• 检查未处理列的数据类型
  remainder='passthrough'会直接保留未指定处理的列，如果这些列里还有其他字符串型分类变量，就会导致模型拟合失败。可以：

  1. 执行print(X.dtypes)找出所有非数值型列，将它们全部加入ColumnTransformer的处理列表；
  2. 若其他列无需保留，可将remainder设为'drop'，只保留编码后的列。

• 验证转换后的特征矩阵
  转换后执行print(X_train.dtype)和print(X_train[:5])，确认所有元素都是数值类型。如果仍存在字符串，说明还有列未被正确编码。

• 修正后的示例代码
  假设州名列索引为0，且其他列均为数值型，修正后的代码如下：

  import numpy as np
  import pandas as pd 
  from sklearn.model_selection import train_test_split
  from sklearn.linear_model import LinearRegression   
  from sklearn.compose import ColumnTransformer
  from sklearn.preprocessing import OneHotEncoder
    
  data_set = pd.read_csv('Unemployment in America Per US State.csv')  
  
  # 先确认列信息与数据类型
  print(data_set.columns)
  print(data_set.dtypes)
    
  X = data_set.iloc[:, :-1]
  y = data_set.iloc[:, -1]
  
  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
  
  # 替换为州名所在的真实列索引，添加drop='first'避免多重共线性
  ct = ColumnTransformer(
      transformers=[
          ('one_hot', OneHotEncoder(sparse_output=False, drop='first'), [0])
      ],
      remainder='passthrough'
  )
  X_train = ct.fit_transform(X_train)
  X_test = ct.transform(X_test)
  
  # 验证转换后的数据类型
  print(X_train.dtype)
  
  regressor = LinearRegression()  
  regressor.fit(X_train, y_train)
   
  y_pred = regressor.predict(X_test)
  

内容的提问来源于stack exchange，提问作者Aminah Rashid