XGBoost二分类模型两种SHAP值计算结果不一致问题咨询

我训练了一个用于二分类任务的XGBoost模型，为了对模型预测结果做可解释性分析，引入SHAP生成force plot进行解释。但在使用过程中结果不符合预期：某样本预测概率为0.2292（小于0.5），预期SHAP输出值应小于0，但SHAP库生成的force plot显示输出值为8.12，而直接调用XGBoost原生接口通过pred_contribs=True参数输出的SHAP值生成的force plot结果符合预期。

复现代码如下：

import shap
import xgboost as xgb
import json
from scipy.sparse import load_npz

print('Version of SHAP: {}'.format(shap.__version__))
print('Version of XGBoost: {}'.format(xgb.__version__))

Version of SHAP: 0.39.0
Version of XGBoost: 1.4.1

# Read the data
X = load_npz('test_data.npz')
X_dmatrix = xgb.DMatrix(X)

# Read the selected features
with open('feature_list.json', 'r') as file:
    feature_list = json.load(file)
    
feature_names = [f'Feature {x:04d}' for x in range(len(feature_list))]

# Read the XGBoost model
xgb_model = xgb.Booster()
xgb_model.load_model('xgboost.json')

# Model prediction
model_pred_detail = xgb_model.predict(X_dmatrix, pred_contribs=True)
model_pred_prob = xgb_model.predict(X_dmatrix)
model_pred_detail.shape

(7887, 501)

# Random select a case
xid=4549
print('Predict proba: {:.04f}'.format(model_pred_prob[xid]))

Predict proba: 0.2292

# Doing SHAP way
explainer = shap.Explainer(xgb_model, feature_names=feature_names, algorithm='tree')
shap_values = explainer(X.toarray())

shap.plots.force(shap_values[xid])

shap.force_plot(
    model_pred_detail[xid, -1], # From XGBoost.Booster.predict with pred_contribs=True
    model_pred_detail[xid, 0:-1], # From XGBoost.Booster.predict with pred_contribs=True
    feature_names=feature_names, 
    features=X[xid].toarray()
)

补充说明：因无法公开自有数据，使用Kaggle公开的COVID-19 NLP文本分类数据集复现了该问题，模型训练代码如下，问题仍然存在：

import pandas as pd
import numpy as np
import re
from sklearn.feature_extraction.text import CountVectorizer
import xgboost as xgb
from sklearn.model_selection import train_test_split
import io
from scipy.sparse import save_npz

# parameter setting
class_weight = 10
minNgramLength = 1
maxNgramLength = 3
binary = False
min_df = 20

# Convert to fix the problem of encoding
with open('Corona_NLP_train.csv', 'rb') as file:
    csv_file = file.read()
csv_file2 = csv_file.decode('utf-8', errors='replace')

# Read and split data
df_note = pd.read_csv(io.StringIO(csv_file2), encoding='utf-8')
df_note['label'] = np.where(df_note['Sentiment'].str.contains('negative', flags=re.I), 0, 1)

df_train, df_test = train_test_split(df_note, test_size=0.2, random_state=42)

# Tokenization
vectorizer = CountVectorizer(max_df=0.98,
                             min_df=min_df,
                             binary=binary,
                            ngram_range=(minNgramLength, maxNgramLength))
vectorizer.fit(df_train['OriginalTweet'])
X_train = vectorizer.transform(df_train['OriginalTweet']).astype(float)
y_train = df_train['label'].astype(float).reset_index(drop=True)

last_params ={
 'lambda': 0.00016096144192346114,
 'alpha': 0.057770973181367063,
 'eta': 0.19258319097144733,
 'gamma': 0.40032424821976653,
 'max_depth': 9,
 'min_child_weight': 5,
 'subsample': 0.31304772813494836,
 'colsample_bytree': 0.4214452441229869,
 'objective': 'binary:logistic',
 'verbosity': 0,
 'n_estimators': 400
}

classifierCV = xgb.XGBClassifier(**last_params, importance_type='gain')
classifierCV.fit(X_train, y_train)

# Get the features
feature_names = vectorizer.get_feature_names_out()

# save model
classifierCV.get_booster().save_model('xgboost.json')

# Save features
import json
with open('feature_list.json', 'w') as file:
    file.write(json.dumps({y:x for x, y in enumerate(feature_names)}))

# save data
save_npz('test_data.npz', X_train)

想咨询导致该现象的原因是什么，哪一种SHAP值是解释XGBoost模型的正确结果？

原因说明 #

导致结果不一致的核心原因有两点：

• 你使用的SHAP 0.39.0属于老旧版本，对XGBoost原生Booster对象的输出空间识别存在适配bug，无法正确匹配binary:logistic目标的输出尺度，最终计算的SHAP值尺度完全偏离预期。
• 二者默认输出空间不匹配：XGBoost原生pred_contribs=True返回的SHAP值默认和模型原始输出（对数几率/logit空间）对齐，所有特征SHAP值加和+偏置项等于该样本的logit输出，经过sigmoid转换后就是你拿到的预测概率。而SHAP库的Explainer默认会根据模型类型自动做输出转换，分类场景下如果没有显式指定output_margin=True，会尝试将SHAP值映射到概率空间，旧版本的转换逻辑存在缺陷，进一步放大了结果偏差。

正确结果判断 #

XGBoost原生pred_contribs返回的SHAP值是完全正确的，这是XGBoost官方基于树结构实现的精确SHAP计算逻辑，和模型训练、推理逻辑完全对齐，可直接用于模型解释。
如果要让SHAP库的输出和原生结果对齐，可做两个调整：升级SHAP到0.40及以上版本修复适配bug，同时初始化Explainer时显式指定output_margin=True，强制返回logit空间的SHAP值即可。

内容的提问来源于stack exchange，提问作者Felix Chan