使用KNN填充DataFrame缺失值时触发ValueError问题求助

我在使用KNN算法填充DataFrame缺失值时遇到了问题。以下是df.info()的输出结果：

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1162 entries, 0 to 1161
Data columns (total 61 columns):
 #   Column                                        Non-Null Count  Dtype  
---  ------                                        --------------  -----  
 0   Algorithms&DataStructures                     428 non-null    float64
 1   C/C++Programming                              688 non-null    float64
 2   Calculus1                                     835 non-null    float64
 3   Calculus2                                     752 non-null    float64
 4   Calculus3                                     366 non-null    float64
 5   ChemistryLaboratory                           497 non-null    float64
 6   ChemistryforEngineers                         823 non-null    float64
 7   ComputerArchitecture                          433 non-null    float64

我使用的NaN填充函数来自Kaggle的《Bayesian Regression | House Price Prediction》notebook，代码如下：

from sklearn.neighbors import KNeighborsRegressor
import numpy as np
import pandas as pd

# function that imputes a dataframe 
def impute_knn(df):
    
    ''' inputs: pandas df containing feature matrix '''
    ''' outputs: dataframe with NaN imputed '''
    # imputation with KNN unsupervised method
    # separate dataframe into numerical/categorical
    ldf = df.select_dtypes(include=[np.number])           # select numerical columns in df
    ldf_putaside = df.select_dtypes(exclude=[np.number])  # select categorical columns in df
    # define columns w/ and w/o missing data
    cols_nan = ldf.columns[ldf.isna().any()].tolist()         # columns w/ nan 
    cols_no_nan = ldf.columns.difference(cols_nan).values     # columns w/o nan 

    for col in cols_nan:                
        imp_test = ldf[ldf[col].isna()]   # indicies which have missing data will become our test set
        imp_train = ldf.dropna()          # all indicies which which have no missing data 
        model = KNeighborsRegressor(n_neighbors=5)  # KNR Unsupervised Approach
        knr = model.fit(imp_train[cols_no_nan], imp_train[col])
        ldf.loc[df[col].isna(), col] = knr.predict(imp_test[cols_no_nan])
    
    return pd.concat([ldf,ldf_putaside],axis=1)

调用该函数时触发了以下错误：

ValueError                                Traceback (most recent call last)
<ipython-input-284-b13fac408835> in <module>
----> 1 df2 = impute_knn(df)
      2 # looks like we have a full feature matrix
      3 df2.info()

5 frames
/usr/local/lib/python3.8/dist-packages/sklearn/utils/validation.py in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator)
    663 
    664         if all(isinstance(dtype, np.dtype) for dtype in dtypes_orig):
--> 665             dtype_orig = np.result_type(*dtypes_orig)
    666 
    667     if dtype_numeric:

<__array_function__ internals> in result_type(*args, **kwargs)

ValueError: at least one array or dtype is required

错误核心原因 #

这个报错的本质是cols_no_nan为空数组——也就是所有数值列都存在缺失值。原函数逻辑中，训练KNN模型时依赖完全无缺失的列作为特征，但当所有列都有缺失时，imp_train[cols_no_nan]会返回空DataFrame，而KNeighborsRegressor.fit()无法处理空的特征矩阵，进而触发np.result_type的参数错误。

从你提供的df.info()结果能看到，列出的8个数值列非空计数均小于总行数1162，说明所有数值列都存在缺失值，直接导致cols_no_nan没有任何元素。

修复方案 #

方案1：调整KNN填充逻辑，适配全列有缺失的场景

修改函数逻辑，对每个要填充的列，使用除当前列外的所有数值列作为特征，同时过滤掉特征列存在缺失的行来构建训练集和测试集：

from sklearn.neighbors import KNeighborsRegressor
import numpy as np
import pandas as pd

def impute_knn(df):
    ldf = df.select_dtypes(include=[np.number])
    ldf_putaside = df.select_dtypes(exclude=[np.number])
    cols_nan = ldf.columns[ldf.isna().any()].tolist()

    for col in cols_nan:
        # 取当前列之外的所有数值列作为特征
        feature_cols = [c for c in ldf.columns if c != col]
        # 训练集：当前列和特征列均无缺失的行
        imp_train = ldf.dropna(subset=[col] + feature_cols)
        # 测试集：当前列缺失但特征列无缺失的行
        imp_test = ldf[ldf[col].isna()].dropna(subset=feature_cols)
        
        # 处理训练集或测试集为空的情况，用均值填充剩余缺失值
        if len(imp_train) == 0 or len(imp_test) == 0:
            ldf.loc[ldf[col].isna(), col] = ldf[col].mean()
            continue
            
        model = KNeighborsRegressor(n_neighbors=5)
        knr = model.fit(imp_train[feature_cols], imp_train[col])
        ldf.loc[imp_test.index, col] = knr.predict(imp_test[feature_cols])
        
        # 处理测试集中特征列也有缺失的行，用均值填充
        remaining_nan = ldf[col].isna() & ldf[feature_cols].isna().any(axis=1)
        ldf.loc[remaining_nan, col] = ldf[col].mean()
    
    return pd.concat([ldf, ldf_putaside], axis=1)

方案2：先简单填充部分列，再用原逻辑运行KNN

如果想保留原函数逻辑，可以先对缺失率较低的列用均值/中位数填充，制造出完全无缺失的特征列后再运行原函数：

# 先填充缺失率低于20%的列
low_missing_cols = ldf.columns[ldf.isna().mean() < 0.2]
ldf[low_missing_cols] = ldf[low_missing_cols].fillna(ldf[low_missing_cols].mean())

# 再运行原impute_knn函数
df_filled = impute_knn(ldf)

额外注意事项 #

• 运行KNN填充前，建议对数值列做标准化/归一化，因为KNN基于距离计算，不同尺度的特征会严重影响填充结果。
• 如果某列缺失率超过50%，建议考虑删除该列，或者使用MICE等更适合高缺失率场景的填充方法。

内容的提问来源于stack exchange，提问作者TruongQuocAn