初学者使用Naive Bayes分类时遇ValueError错误求助

问题描述 #

无编程基础的大学生在项目中使用BernoulliNB（伯努利朴素贝叶斯）做分类时，运行代码出现以下错误：

ValueError: Input contains NaN, infinity or a value too large for dtype('float64')

无法理解现有论坛解决方案，附上代码及报错栈请求帮助。

原代码 #

dat = pd.get_dummies(df)

# Define X and y
X = dat.drop('RESP', axis = 1)
y = dat['RESP']

# training and testing data
from sklearn.model_selection import train_test_split

# assign test data size 30%
X_train, X_test, y_train, y_test =train_test_split(X,y,test_size= 0.3, random_state=0)

#BERNOULLI
# importing classifier
from sklearn.naive_bayes import BernoulliNB

# initializaing the NB
classifer = BernoulliNB()

# training the model
classifer.fit(X_train, y_train)

# testing the model
y_pred = classifer.predict(X_test)

报错栈 #

---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-22-df3c037d90d0> in <module>
     23 
     24 # testing the model
---&gt; 25 y_pred = classifer.predict(X_test)

5 frames

/usr/local/lib/python3.8/dist-packages/sklearn/naive_bayes.py in predict(self, X)
     80         """
     81         check_is_fitted(self)
---&gt; 82         X = self._check_X(X)
     83         jll = self._joint_log_likelihood(X)
     84         return self.classes_[np.argmax(jll, axis=1)]

/usr/local/lib/python3.8/dist-packages/sklearn/naive_bayes.py in _check_X(self, X)
   1145     def _check_X(self, X):
   1146         """Validate X, used only in predict* methods."""
-&gt; 1147         X = super()._check_X(X)
   1148         if self.binarize is not None:
   1149             X = binarize(X, threshold=self.binarize)

/usr/local/lib/python3.8/dist-packages/sklearn/naive_bayes.py in _check_X(self, X)
    517     def _check_X(self, X):
    518         """Validate X, used only in predict* methods."""
--&gt; 519         return self._validate_data(X, accept_sparse="csr", reset=False)
    520 
    521     def _check_X_y(self, X, y, reset=True):

/usr/local/lib/python3.8/dist-packages/sklearn/base.py in _validate_data(self, X, y, reset, validate_separately, **check_params)
    564             raise ValueError("Validation should be done on X, y or both.")
    565         elif not no_val_X and no_val_y:
--&gt; 566             X = check_array(X, **check_params)
    567             out = X
    568         elif no_val_X and not no_val_y:

/usr/local/lib/python3.8/dist-packages/sklearn/utils/validation.py in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator)
    798 
    799         if force_all_finite:
--&gt; 800             _assert_all_finite(array, allow_nan=force_all_finite == "allow-nan")
    801 
    802     if ensure_min_samples &gt; 0:

/usr/local/lib/python3.8/dist-packages/sklearn/utils/validation.py in _assert_all_finite(X, allow_nan, msg_dtype)
    112         ):
    113             type_err = "infinity" if allow_nan else "NaN, infinity"
--&gt; 114             raise ValueError(
    115                 msg_err.format(
    116                     type_err, msg_dtype if msg_dtype is not None else X.dtype

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

解决方案 #

错误核心是输入数据（X_test）存在NaN、无穷大或超出float64范围的异常值，sklearn模型无法处理这类数据，按以下步骤修复：

1. 定位异常值 #

先检查数据中的缺失值和无穷大值：

import numpy as np
import pandas as pd

# 查看处理后数据dat的缺失值分布
print(dat.isnull().sum())
# 查看训练集、测试集的无穷大值数量
print(np.isinf(X_train).sum())
print(np.isinf(X_test).sum())

2. 处理缺失值（NaN） #

根据缺失情况选择合适方法：

• 删除法：若缺失值占比极低，直接删除含缺失值的行或列：

  # 删除所有含NaN的行
  dat = dat.dropna()
  # 删除缺失率超过50%的列
  threshold = len(dat) * 0.5
  dat = dat.drop(columns=[col for col in dat.columns if dat[col].isnull().sum() > threshold])
  

• 填充法：用统计值填充，数值型列用均值/中位数，分类型列用众数：

  # 数值列填充均值
  dat = dat.fillna(dat.mean(numeric_only=True))
  # 分类列填充众数
  dat = dat.fillna(dat.mode().iloc[0])
  # 用sklearn工具统一处理
  from sklearn.impute import SimpleImputer
  imputer = SimpleImputer(strategy='mean')  # 分类数据改用strategy='most_frequent'
  num_cols = dat.select_dtypes(include=['int64','float64']).columns
  dat[num_cols] = imputer.fit_transform(dat[num_cols])
  

3. 处理无穷大值 #

将无穷大值替换为NaN后，按缺失值处理：

# 替换inf和-inf为NaN
dat = dat.replace([np.inf, -np.inf], np.nan)
# 填充替换后的NaN
dat = dat.fillna(dat.mean(numeric_only=True))

4. 重新运行代码 #

处理完所有异常值后，重新执行原代码中的数据集拆分、模型训练和预测步骤即可。

内容的提问来源于stack exchange，提问作者Sebastian Sarna