从转成Numpy数组的Pandas数据框删除空值列时遇KeyError求助

问题背景 #

我尝试用K近邻回归模型完成预测任务，先构建了训练集和测试集的Pandas数据框：

import numpy as np
import pandas as pd
from sklearn.neighbors import KNeighborsRegressor

############
### DATA ###
############
TrainingData = { 'name':  ['Alex', 'Ben', 'Marry','Alex', 'Ben', 'Marry'],
                'teacher': [1,0,0,1,0,0],
                'doctor': [0,1,0,0,1,0],
                'engineer': [0,0,1,0,0,1],
                'age': [27, 32, 78,27, 32, 78],
                'weight': [160, 209, 130,164, 206, 132],
                'date': [1,1,1,2,2,2]}

TestData = {'name': ['Alex', 'Ben', 'Marry'],
            'teacher': [1,0,0],
            'doctor': [0,1,0],
            'engineer': [0,0,1],
            'age': [np.NaN,np.NaN,np.NaN],
            'weight': [np.NaN,np.NaN,np.NaN],
            'data': [3,3,3]}
# Convert to pandas dataframe
dfTraining = pd.DataFrame(TrainingData)
dfTest = pd.DataFrame(TestData)

训练集输出：

name  teacher  doctor  engineer  age  weight  date
0   Alex        1       0         0   27     160     1
1    Ben        0       1         0   32     209     1
2  Marry        0       0         1   78     130     1
3   Alex        1       0         0   27     164     2
4    Ben        0       1         0   32     206     2
5  Marry        0       0         1   78     132     2

测试集输出：

name  teacher  doctor  engineer  age  weight  data
0   Alex        1       0         0  NaN     NaN     3
1    Ben        0       1         0  NaN     NaN     3
2  Marry        0       0         1  NaN     NaN     3

随后我尝试转换数据适配模型：

Y=dfTraining.groupby('name')['weight'].apply(lambda x: (x.to_numpy()))
df_train_x=dfTraining.drop('weight', axis=1)
X= df_train_x.groupby('name').apply(lambda x: (x.to_numpy()))

K=1

df_test_x=dfTest.drop('weight', axis=1)
X_pred_null=df_test_x.groupby('name').apply(lambda x: (x.notnull()))
PresentVariables = (X_pred_null.to_numpy())

执行训练代码时触发KeyError：

NearestNeighbor = KNeighborsRegressor(n_neighbors=K).fit(X[:, PresentVariables[0]], Y)

错误信息：

KeyError                                  Traceback (most recent call last)
Input In [23], in <cell line: 1>()
----> 1 NearestNeighbor = KNeighborsRegressor(n_neighbors=K).fit(X[:, PresentVariables[0]], Y)

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/pandas/core/series.py:906, in Series.__getitem__(self, key)
    903     key = np.asarray(key, dtype=bool)
    904     return self._get_values(key)
--> 906 return self._get_with(key)

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/pandas/core/series.py:921, in Series._get_with(self, key)
    916     raise TypeError(
    917         "Indexing a Series with DataFrame is not "
    918         "supported, use the appropriate DataFrame column"
    919     )
    920 elif isinstance(key, tuple):
--> 921     return self._get_values_tuple(key)
    923 elif not is_list_like(key):
    924     # e.g. scalars that aren't recognized by lib.is_scalar, GH#32684
    925     return self.loc[key]

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/pandas/core/series.py:956, in Series._get_values_tuple(self, key)
    953     return result
    955 if not isinstance(self.index, MultiIndex):
--> 956     raise KeyError("key of type tuple not found and not a MultiIndex")
    958 # If key is contained, would have returned by now
    959 indexer, new_index = self.index.get_loc_level(key)

KeyError: 'key of type tuple not found and not a MultiIndex'

错误原因 #

1. X的类型错误：groupby.apply返回的是Pandas Series，而非二维Numpy数组，无法使用Numpy的[:, ...]索引方式，这是触发KeyError的直接原因。
2. 数据结构不符合KNN要求：Sklearn的KNeighborsRegressor.fit()要求输入的特征矩阵是二维数组（样本数×特征数），目标数组是一维数组（每个样本对应一个目标值），按name分组后的数据结构完全不符合该要求。
3. 空值列判断逻辑冗余：无需通过分组后的notnull结果筛选列，直接基于测试集的非空列即可确定训练集要保留的特征。

解决方案 #

步骤1：统一数据列名（修正笔误） #

测试集列名data与训练集date不一致，先统一：

dfTest = dfTest.rename(columns={'data': 'date'})

步骤2：正确拆分训练集特征与目标 #

直接提取符合KNN要求的特征矩阵和目标数组：

# 训练集特征：排除name和weight列
X_train = dfTraining.drop(['name', 'weight'], axis=1).values
# 训练集目标：weight列
y_train = dfTraining['weight'].values

步骤3：筛选非空特征列 #

从测试集中确定无空值的特征列，同步筛选训练集特征：

# 测试集特征列（排除name和weight）
test_features = dfTest.drop(['name', 'weight'], axis=1)
# 获取全量非空列的布尔索引
non_null_cols = test_features.notnull().all(axis=0)
# 筛选训练集对应的非空特征列
X_train_filtered = X_train[:, non_null_cols.values]

步骤4：训练模型并预测 #

用筛选后的特征训练模型，完成测试集预测：

K = 1
# 初始化并训练模型
knn = KNeighborsRegressor(n_neighbors=K)
knn.fit(X_train_filtered, y_train)

# 处理测试集特征（保留相同的非空列）
X_test_filtered = test_features.loc[:, non_null_cols].values
# 执行预测
y_pred = knn.predict(X_test_filtered)

# 将预测结果合并到测试集
dfTest['predicted_weight'] = y_pred
print(dfTest)

最终输出 #

name  teacher  doctor  engineer  age  weight  date  predicted_weight
0   Alex        1       0         0  NaN     NaN     3              162.0
1    Ben        0       1         0  NaN     NaN     3              207.5
2  Marry        0       0         1  NaN     NaN     3              131.0

补充说明 #

如果确实需要按name分组做预测（比如以组为单位提取特征），需将每个组的特征转换为单一向量（如取均值），但常规场景下KNN以单个样本为单位训练更合理。同时必须保证训练集与测试集的特征列严格对应，避免因列名或数量不一致导致的错误。

内容的提问来源于stack exchange，提问作者mnm