Linux正常运行的SHAP代码在Windows10报ValueError求助

问题 #

在Linux系统运行以下SHAP相关代码正常，但Windows 10系统运行时触发ValueError，错误提示：Input contains NaN, infinity or a value too large for dtype('float64')。

运行代码 #

import shap
# 输出JS可视化代码到 notebook
shap.initjs()

def prob(data):
    return model.forward(Variable(torch.from_numpy(data)).float()).detach().numpy().reshape(-1, 1)

shap_explainer = shap.KernelExplainer(prob, 
                                      X_train.numpy(), 
                                      link="logit")
shap_values = shap_explainer.shap_values(X_test.numpy(), nsamples=100)

错误回溯 #

ValueError Traceback (most recent call last)

in
2 X_train.numpy(),
3 link="logit")
----> 4 shap_values = shap_explainer.shap_values(X_test.numpy(), nsamples=100)

D:\Software\anaconda3\lib\site-packages\shap\explainers_kernel.py in shap_values(self, X, **kwargs)
188 if self.keep_index:
189 data = convert_to_instance_with_index(data, column_name, index_value[i:i +1], index_name)
--> 190 explanations.append(self.explain(data, **kwargs))
191 if kwargs.get("gc_collect", False):
192 gc.collect()

D:\Software\anaconda3\lib\site-packages\shap\explainers_kernel.py in explain(self, incoming_instance, **kwargs)
386 phi_var = np.zeros((self.data.groups_size, self.D))
387 for d in range(self.D):
--> 388 vphi, vphi_var = self.solve(self.nsamples / self.max_samples, d)
389 phi[self.varyingInds, d] = vphi
390 phi_var[self.varyingInds, d] = vphi_var

D:\Software\anaconda3\lib\site-packages\shap\explainers_kernel.py in solve(self, fraction_evaluated, dim)
563 elif self.l1_reg == "auto" or self.l1_reg == "bic" or self.l1_reg == "aic":
564 c = "aic" if self.l1_reg == "auto" else self.l1_reg
--> 565 nonzero_inds = np.nonzero(LassoLarsIC(criterion=c).fit(mask_aug, eyAdj_aug).coef_)[0]
566
567 # use a fixed regularization coeffcient

D:\Software\anaconda3\lib\site-packages\sklearn\linear_model_least_angle.py in fit(self, X, y, copy_X)
1862 if copy_X is None:
1863 copy_X = self.copy_X
--> 1864 X, y = self._validate_data(X, y, y_numeric=True)
1865
1866 X, y, Xmean, ymean, Xstd = LinearModel._preprocess_data(

D:\Software\anaconda3\lib\site-packages\sklearn\base.py in _validate_data(self, X, y, reset, validate_separately, **check_params)
431 y = check_array(y, **check_y_params)
432 else:
--> 433 X, y = check_X_y(X, y, **check_params)
434 out = X, y
435

D:\Software\anaconda3\lib\site-packages\sklearn\utils\validation.py in inner_f(*args, **kwargs)
61 extra_args = len(args) - len(all_args)
62 if extra_args <= 0:
--> 63 return f(*args, **kwargs)
64
65 # extra_args > 0

D:\Software\anaconda3\lib\site-packages\sklearn\utils\validation.py in check_X_y(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)
825 else:
826 y = column_or_1d(y, warn=True)
--> 827 _assert_all_finite(y)
828 if y_numeric and y.dtype.kind == 'O':
829 y = y.astype(np.float64)

D:\Software\anaconda3\lib\site-packages\sklearn\utils\validation.py in _assert_all_finite(X, allow_nan, msg_dtype)
101 not allow_nan and not np.isfinite(X).all()):
102 type_err = 'infinity' if allow_nan else 'NaN, infinity'
--> 103 raise ValueError(
104 msg_err.format
105 (type_err,

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

解决方案 #

1. 排查输入数据异常 #

先验证X_train和X_test是否存在NaN、无穷大或超出float64范围的值，添加以下代码检查：

import numpy as np

# 检查训练集
print("X_train NaN数量:", np.isnan(X_train.numpy()).sum())
print("X_train 无穷大数量:", np.isinf(X_train.numpy()).sum())
print("X_train 超出float64范围:", np.any(np.abs(X_train.numpy()) > np.finfo(np.float64).max))

# 检查测试集
print("X_test NaN数量:", np.isnan(X_test.numpy()).sum())
print("X_test 无穷大数量:", np.isinf(X_test.numpy()).sum())
print("X_test 超出float64范围:", np.any(np.abs(X_test.numpy()) > np.finfo(np.float64).max))

若发现异常值，用均值/中位数填充或删除对应样本处理。

2. 验证模型输出有效性 #

问题可能出在prob函数返回的预测值中，修改函数添加异常检测：

def prob(data):
    output = model.forward(Variable(torch.from_numpy(data)).float()).detach().numpy().reshape(-1, 1)
    # 检查输出是否有异常
    if np.isnan(output).any() or np.isinf(output).any():
        print("模型输出存在NaN/无穷大")
        print("异常输出:", output[np.isnan(output) | np.isinf(output)])
    return output

如果模型输出异常，排查Windows环境下的模型权重加载是否完整，或是否存在平台相关的数值计算差异。

3. 调整SHAP正则化参数 #

Windows下数值精度差异可能导致LassoLarsIC拟合失败，显式指定l1_reg替代默认的auto：

shap_explainer = shap.KernelExplainer(prob, 
                                      X_train.numpy(), 
                                      link="logit",
                                      l1_reg=0.01)  # 可根据数据调整系数

测试阶段也可尝试关闭正则化（不推荐用于正式场景）：l1_reg=None。

4. 统一数据类型精度 #

确保所有数据 dtype 统一为float64，避免类型转换带来的精度问题：

X_train = X_train.astype(np.float64)
X_test = X_test.astype(np.float64)

5. 降低采样数量 #

减小nsamples值，减少计算量和数值不稳定概率：

shap_values = shap_explainer.shap_values(X_test.numpy(), nsamples=50)

内容的提问来源于stack exchange，提问作者Frank