使用MultiOutputClassifier执行RFECV时出现NaN值错误求助

我尝试使用多输出模型执行递归特征消除（Recursive Feature Elimination with Cross-Validation，RFECV），以下是最小可复现代码：

from sklearn.multioutput import MultiOutputClassifier
import numpy as np 
import pandas as pd 
from catboost import CatBoostClassifier 
from sklearn.pipeline import Pipeline, make_pipeline 
from sklearn.metrics import accuracy_score, make_scorer 
from sklearn.preprocessing import StandardScaler 
from sklearn.feature_selection import RFECV 
from sklearn.model_selection import cross_validate

data = {"salary": [60000, 50000, 10000, 94000, 34000, 70000], "age": [23, 27, 13, 45, 60, 70], "weight": [80, 90, 50, 75, 60, 70], "height": [90, 100, 70, 60, 50, 70], }

df = pd.DataFrame(data=data) 

y = np.asarray([[1, 0, 1], [0, 1, 0], [1, 1, 0], [1, 0, 0], [0, 1, 0], [1, 0, 1]])

model = MultiOutputClassifier(CatBoostClassifier(verbose=0)) 

pipe = Pipeline(steps = [("preprocessor", StandardScaler()), ("estimator", model)])

rfe_cv_pipe = make_pipeline(StandardScaler(), RFECV(estimator=model, scoring=make_scorer(accuracy_score), min_features_to_select=2, importance_getter="estimator.feature_importances_", cv=2))

scores = cross_validate(pipe, df, y, cv=2, return_train_score=True) 

rfe_cv_pipe.fit(df, y)

使用cross_validate测试pipe可以正常得到训练和验证分数，但调用rfe_cv_pipe.fit()时出现错误：输入包含NaN、无穷大或超出dtype('float64')范围的值，完整错误栈如下：

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/name/miniconda3/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/Users/name/miniconda3/lib/python3.10/site-packages/sklearn/pipeline.py", line 427, in fit
    self._final_estimator.fit(Xt, y, **fit_params_last_step)
  File "/Users/name/miniconda3/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/Users/name/miniconda3/lib/python3.10/site-packages/sklearn/feature_selection/_rfe.py", line 726, in fit
    scores = parallel(
  File "/Users/name/miniconda3/lib/python3.10/site-packages/sklearn/feature_selection/_rfe.py", line 727, in <genexpr>
    func(rfe, self.estimator, X, y, train, test, scorer)
  File "/Users/name/miniconda3/lib/python3.10/site-packages/sklearn/feature_selection/_rfe.py", line 32, in _rfe_single_fit
    return rfe._fit(
  File "/Users/name/miniconda3/lib/python3.10/site-packages/sklearn/feature_selection/_rfe.py", line 300, in _fit
    importances = _get_feature_importances(
  File "/Users/name/miniconda3/lib/python3.10/site-packages/sklearn/feature_selection/_base.py", line 239, in _get_feature_importances
    importances = safe_sqr(importances).sum(axis=0)
  File "/Users/name/miniconda3/lib/python3.10/site-packages/sklearn/utils/__init__.py", line 712, in safe_sqr
    X = check_array(X, accept_sparse=["csr", "csc", "coo"], ensure_2d=False)
  File "/Users/name/miniconda3/lib/python3.10/site-packages/sklearn/utils/validation.py", line 957, in check_array
    _assert_all_finite(
  File "/Users/name/miniconda3/lib/python3.10/site-packages/sklearn/utils/validation.py", line 122, in _assert_all_finite
    _assert_all_finite_element_wise(
  File "/Users/name/miniconda3/lib/python3.10/site-packages/sklearn/utils/validation.py", line 171, in _assert_all_finite_element_wise
    raise ValueError(msg_err)
ValueError: Input contains NaN.

核心原因 #

MultiOutputClassifier会为每个输出维度单独训练一个子模型，它的feature_importances_是二维数组（形状为(n_outputs, n_features)），而RFECV默认期望输入一维的特征重要性数组。当sklearn尝试对二维数组执行平方求和操作时，若子模型因样本量过小等原因训练不稳定，就会产生NaN值触发错误。

另外你的数据集仅包含6个样本，设置cv=2会让每个交叉验证折叠只有3个样本，CatBoost在极小样本下训练容易出现特征重要性计算失效的情况，进一步加剧NaN问题。

修复方案 #

1. 自定义特征重要性聚合逻辑
   修改RFECV的importance_getter参数，将多个子模型的特征重要性聚合为一维数组，比如取所有输出维度的均值：

   rfe_cv_pipe = make_pipeline(
       StandardScaler(), 
       RFECV(
           estimator=model, 
           scoring=make_scorer(accuracy_score), 
           min_features_to_select=2, 
           # 聚合所有子模型的特征重要性为均值
           importance_getter=lambda clf: np.mean([est.feature_importances_ for est in clf.estimators_], axis=0),
           cv=3  # 调整CV折数，减少单折叠样本量过小的问题
       )
   )
   

2. 优化数据集或CV设置
   • 尽可能增加样本数量，避免小样本导致的模型训练不稳定；
   • 调整cv参数为更大的值（比如3），让每个折叠的样本量更合理。

内容的提问来源于stack exchange，提问作者Caesar