拟合Logistic回归模型时遇ValueError报错求助

问题描述 #

调用model.fit(X_train, y_train)时持续报错：

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

已做排查：

• 用isnull检测空值，无异常
• 检测无穷值，无异常
• y值最大为10493，确认未超出范围

完整报错堆栈：

ValueError                                Traceback (most recent call last)
/var/folders/pt/w87p29y92_z976_m8dv5djq00000gq/T/ipykernel_8727/2148996027.py in <module>
      1 model = LogisticRegression(max_iter=1000)
----> 2 model.fit(X_train, y_train)

~/opt/anaconda3/lib/python3.9/site-packages/sklearn/linear_model/_logistic.py in fit(self, X, y, sample_weight)
   1506             _dtype = [np.float64, np.float32]
   1507 
-> 1508         X, y = self._validate_data(
   1509             X,
   1510             y,

~/opt/anaconda3/lib/python3.9/site-packages/sklearn/base.py in _validate_data(self, X, y, reset, validate_separately, **check_params)
    579                 y = check_array(y, **check_y_params)
    580             else:
--> 581                 X, y = check_X_y(X, y, **check_params)
    582             out = X, y
    583 

~/opt/anaconda3/lib/python3.9/site-packages/sklearn/utils/validation.py in check_X_y(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)
    962         raise ValueError("y cannot be None")
    963 
--> 964     X = check_array(
    965         X,
    966         accept_sparse=accept_sparse,

~/opt/anaconda3/lib/python3.9/site-packages/sklearn/utils/validation.py in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator)
    798 
    799         if force_all_finite:
--> 800             _assert_all_finite(array, allow_nan=force_all_finite == "allow-nan")
    801 
    802     if ensure_min_samples > 0:

~/opt/anaconda3/lib/python3.9/site-packages/sklearn/utils/validation.py in _assert_all_finite(X, allow_nan, msg_dtype)
    112         ):
    113             type_err = "infinity" if allow_nan else "NaN, infinity"
--> 114             raise ValueError(
    115                 msg_err.format(
    116                     type_err, msg_dtype if msg_dtype is not None else X.dtype

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

解决方法 #

1. 精准排查异常值 #

用numpy工具彻底扫描数据集，避免遗漏隐性异常：

import numpy as np

# 检查X_train的异常值
print("X_train NaN总数:", np.isnan(X_train).sum().sum())
print("X_train 正无穷总数:", np.isposinf(X_train).sum().sum())
print("X_train 负无穷总数:", np.isneginf(X_train).sum().sum())

# 检查y_train的异常值
print("y_train NaN总数:", np.isnan(y_train).sum())
print("y_train 无穷值总数:", np.isinf(y_train).sum())

如果是DataFrame，也可以用X_train.replace([np.inf, -np.inf], np.nan).isnull().sum().sum()统计所有异常值总和。

2. 检查数据类型一致性 #

若数据存在非数值类型列（如字符串、混合类型），转换为float时可能生成隐性NaN，先检查每列数据类型：

print(X_train.dtypes)

对非数值列做编码处理或直接剔除，确保所有特征为数值型。

3. 处理X_train中的极端大值 #

y值正常不代表X特征没问题，float64的上限约为1.8e308，若X中存在超出该范围的值会触发报错。先查看特征极值：

print("X_train各列最大值:\n", X_train.max())
print("X_train各列最小值:\n", X_train.min())

若存在过大值，对特征做标准化处理：

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)

用处理后的X_train_scaled重新训练模型。

4. 验证分割后的数据集 #

如果用train_test_split拆分数据，需单独检查训练集，避免整体数据正常但训练集出现异常的情况。

5. 用sklearn工具定位异常位置 #

直接调用sklearn的验证函数定位具体异常点：

from sklearn.utils.validation import check_array

try:
    check_array(X_train, force_all_finite=True)
except ValueError:
    # 输出异常值的位置坐标
    print("X_train异常值位置:", np.where(np.isnan(X_train) | np.isinf(X_train)))

内容的提问来源于stack exchange，提问作者Dennis