机器学习新手求教：梯度提升树（GBT）多分类的弱学习器问题

Hey there! It makes total sense that you're confused—most intro GBT resources fixate on regression, and the jump to multi-class classification isn't always spelled out clearly. Let's break down how scikit-learn's GradientBoostingClassifier uses regression trees to handle multi-class tasks, and the core mechanics behind it.

Core Concept: GBT for Classification = Fitting "Error Signals" with Regression Trees #

At its heart, gradient boosting is all about iteratively correcting the mistakes of previous models by fitting the negative gradient of the loss function (think of this as an "error signal" that tells us how to adjust predictions to reduce loss). For classification tasks—including multi-class—we don't change the weak learner (we still use regression trees); instead, we adapt the loss function and how we interpret the tree outputs.

How scikit-learn's GradientBoostingClassifier Works for Multi-Class Tasks #

Let's walk through the step-by-step process, using a K-class classification problem as an example:

1. Initialization #

The model starts with a simple baseline prediction: for each class, it uses the proportion of samples belonging to that class in the training set (the prior probability). This gives us an initial set of raw predictions (before converting to probabilities) for each class.

2. Iterative Tree Training & Prediction Updates #

For each iteration (each new tree added):

• Calculate the negative gradient: For every sample, we compute how much our current predictions are off—this is the negative gradient of the multi-class log loss function. For a sample that belongs to class c, the negative gradient for class c is 1 - p_c (where p_c is the current predicted probability of class c), and for all other classes k ≠ c, it's -p_k. These values are continuous, which is why regression trees (which output continuous values) are perfect for fitting them.
• Train regression trees for each class: We train K separate regression trees, one for each class. Each tree learns to predict the negative gradient values for its corresponding class.
• Update raw predictions: For each class, we take the output of its regression tree, multiply it by a small learning rate (to prevent overfitting via shrinkage), and add this to the class's raw predictions from previous iterations.

3. Final Probability Conversion #

After all iterations are done, we take the raw predictions for each class and pass them through the softmax function. This converts the raw values into probabilities that sum to 1, and the class with the highest probability is our final prediction.

Why Regression Trees Instead of Classification Trees? #

Great question! Regression trees are used here because:

• GBT relies on fitting continuous error signals (negative gradients), not discrete class labels. Classification trees only output discrete classes, which can't capture the fine-grained adjustments needed to minimize the loss function.
• Regression trees can produce continuous predictions that directly map to the gradient values, allowing the model to make incremental, precise updates to its predictions each iteration.

This approach keeps the weak learner simple (no need to modify tree structure) while adapting the rest of the algorithm to handle multi-class classification.

内容的提问来源于stack exchange，提问作者Mei Lie