Great question! Let's break this down into three clear parts: first the mathematical logic behind how gradient boosting calculates predicted probabilities, then how to extract per-sample feature contributions, and finally practical Python implementations you can use right away.

1. Mathematical Breakdown of Probability Prediction #

Gradient boosting for classification works by building a sequence of decision trees, each correcting the errors of the previous ones. Here's the step-by-step math for binary classification (the most common case; multi-class extends this):

• Initial Prediction: We start with a baseline log-odds (log of the odds ratio p/(1-p)). For log loss (the standard loss for classification), this baseline is calculated as log(positive_samples / negative_samples) from the training data. Let's call this f₀(x).
• Tree Sequences: Each subsequent tree Tₘ(x) learns to predict the residual (the difference between the true log-odds and the current prediction). We add this tree's output (scaled by a learning rate η) to the previous prediction:

  fₘ(x) = fₘ₋₁(x) + η * Tₘ(x)
  

• Final Probability: After training all M trees, we convert the total log-odds f_M(x) to a probability using the sigmoid function (the inverse of log-odds):

  p(x) = 1 / (1 + exp(-f_M(x)))
  

For multi-class classification, gradient boosting trains a separate set of trees for each class. The final probabilities are computed using the softmax function, which normalizes the log-odds of each class into a sum-to-1 probability distribution.

2. Extracting Per-Sample Feature Contributions #

Feature contributions tell you exactly how much each feature moves the prediction from the baseline probability to the final predicted probability for a single test sample. The two most reliable ways to get these are:

• SHAP Values: A game-theoretic approach that provides consistent, interpretable feature contributions for any tree-based model. SHAP values explain how much each feature increases or decreases the predicted probability relative to the model's average prediction.
• Model-Builtin Methods: Libraries like XGBoost have native support for calculating feature contributions directly, which is faster for large datasets.

3. Python Implementation Examples #

Let's use the breast cancer dataset (built into scikit-learn) for our examples—it's a clean binary classification task.

Example 1: Scikit-Learn GradientBoostingClassifier with SHAP #

SHAP works seamlessly with scikit-learn's gradient boosting model and gives you both numerical contributions and visualizations:

import shap
import numpy as np
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

# Load and split data
data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42
)

# Train the gradient boosting model
gb_model = GradientBoostingClassifier(
    n_estimators=100, learning_rate=0.1, random_state=42
)
gb_model.fit(X_train, y_train)

# Initialize SHAP tree explainer
explainer = shap.TreeExplainer(gb_model)
shap_values = explainer.shap_values(X_test)

# For binary classification, shap_values has two arrays (one per class)
# We'll focus on class 1 (malignant) contributions
shap_class1 = shap_values[1]

# Print contributions for the first test sample
print("Feature contributions for first test sample (class 1 probability):")
for name, contrib in zip(data.feature_names, shap_class1[0]):
    print(f"{name}: {contrib:.4f}")

# Visualize the prediction breakdown (run in a Jupyter notebook for interactive plot)
shap.initjs()
shap.force_plot(
    explainer.expected_value[1],  # Baseline log-odds for class 1
    shap_class1[0],              # Contributions for the sample
    X_test[0],                   # Sample features
    feature_names=data.feature_names
)

Example 2: XGBoost with Built-in Contribution Calculation #

XGBoost lets you directly predict feature contributions using the pred_contribs=True parameter. These contributions correspond directly to each feature's impact on the log-odds:

import xgboost as xgb
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

# Load and prepare data
data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42
)

# Convert to XGBoost's DMatrix format
dtrain = xgb.DMatrix(X_train, label=y_train, feature_names=data.feature_names)
dtest = xgb.DMatrix(X_test, feature_names=data.feature_names)

# Train XGBoost model
params = {
    "objective": "binary:logistic",
    "learning_rate": 0.1,
    "random_state": 42
}
xgb_model = xgb.train(params, dtrain, num_boost_round=100)

# Get feature contributions (includes bias term as last column)
contribs = xgb_model.predict(dtest, pred_contribs=True)

# Print contributions for the first test sample
print("\nXGBoost feature contributions for first test sample:")
for name, contrib in zip(data.feature_names, contribs[0][:-1]):
    print(f"{name}: {contrib:.4f}")
print(f"Bias term: {contribs[0][-1]:.4f}")

# Verify the calculation: sum of contributions = log-odds → sigmoid gives probability
log_odds = contribs[0].sum()
calculated_prob = 1 / (1 + np.exp(-log_odds))
print(f"\nCalculated probability: {calculated_prob:.4f}")
print(f"Model's predicted probability: {xgb_model.predict(dtest)[0]:.4f}")

Key Notes #

• SHAP values are more universally applicable (works with LightGBM, CatBoost, etc.) and come with strong theoretical guarantees for interpretability.
• XGBoost's built-in method is faster and more memory-efficient for large datasets, but is specific to XGBoost.
• Positive contributions increase the predicted probability, negative contributions decrease it.

内容的提问来源于stack exchange，提问作者Mrinal Mahajan