Python中能否在决策树回归的叶节点内执行多元回归？

Great question! Let's tackle each part of your query one by one.

1. Can we replace leaf node mean with multiple regression? #

Absolutely! Traditional regression trees use the mean of the target variable in each leaf as the default prediction, but swapping this with a multiple linear regression model (using the features of samples in the leaf as predictors) is a valid and powerful approach.

This method lets you capture local linear relationships within each leaf that a simple mean might overlook. For example, if samples in a leaf still show a clear linear trend between certain features and the target, fitting a linear regression here will produce more accurate predictions than just averaging the target values.

Just watch out for overfitting: if a leaf has too few samples, the linear regression model will be unstable and prone to noise. You can fix this by setting a minimum sample threshold for leaves (like min_samples_leaf in scikit-learn's DecisionTreeRegressor).

2. Python tools for implementing linear regression in tree leaves (similar to the paper) #

There's no out-of-the-box scikit-learn function that directly replicates the logic from Employing Linear Regression in Regression Tree Leaves, but it's straightforward to build this yourself using existing scikit-learn tools. Here's a step-by-step implementation:

from sklearn.tree import DecisionTreeRegressor
from sklearn.linear_model import LinearRegression
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
import numpy as np
from sklearn.metrics import mean_squared_error

# Load sample regression data
X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Step 1: Train a decision tree to split data into leaves
tree = DecisionTreeRegressor(max_depth=3, min_samples_leaf=20, random_state=42)
tree.fit(X_train, y_train)

# Get leaf IDs for training and test samples
train_leaf_ids = tree.apply(X_train)
test_leaf_ids = tree.apply(X_test)

# Step 2: Fit a linear regression model for each leaf
leaf_models = {}
for leaf_id in np.unique(train_leaf_ids):
    # Filter samples belonging to this leaf
    leaf_mask = train_leaf_ids == leaf_id
    X_leaf = X_train[leaf_mask]
    y_leaf = y_train[leaf_mask]
    
    # Fit linear regression on the leaf's samples
    lr_model = LinearRegression()
    lr_model.fit(X_leaf, y_leaf)
    leaf_models[leaf_id] = lr_model

# Step 3: Predict using the corresponding leaf's linear model
y_pred = []
for leaf_id, sample in zip(test_leaf_ids, X_test):
    pred = leaf_models[leaf_id].predict(sample.reshape(1, -1))[0]
    y_pred.append(pred)
y_pred = np.array(y_pred)

# Evaluate performance
print(f"Test MSE: {mean_squared_error(y_test, y_pred):.2f}")

For more optimized implementations, you could also:

• Customize gradient boosting frameworks like XGBoost or LightGBM to use linear regression in leaves (this requires writing custom objective functions, which is more advanced)
• Adapt this logic for large datasets using pyspark.ml, but the core idea remains the same: split data with a tree, then fit linear models per leaf.

3. Executing multiple regression within a decision tree framework #

Yes, you can absolutely run multiple regression (using multiple features as predictors) within a decision tree regression setup. The example above already does this: each leaf's linear regression model uses all input features to predict the target variable.

If you're referring to multi-output regression (predicting multiple target variables at once), you can extend the approach by fitting a multi-output linear regression model for each leaf. Here's a quick snippet for that case:

from sklearn.multioutput import MultiOutputRegressor

# Simulate a 2-dimensional target variable
y_multi = np.column_stack([y, y * 1.2])

# Train tree to split multi-output data
tree_multi = DecisionTreeRegressor(max_depth=3, min_samples_leaf=20, random_state=42)
tree_multi.fit(X_train, y_multi)
train_leaf_ids_multi = tree_multi.apply(X_train)

# Fit multi-output linear regression per leaf
leaf_multi_models = {}
for leaf_id in np.unique(train_leaf_ids_multi):
    leaf_mask = train_leaf_ids_multi == leaf_id
    X_leaf = X_train[leaf_mask]
    y_leaf = y_multi[leaf_mask]
    
    multi_lr = MultiOutputRegressor(LinearRegression())
    multi_lr.fit(X_leaf, y_leaf)
    leaf_multi_models[leaf_id] = multi_lr

# Predict multiple target variables
y_multi_pred = []
for leaf_id, sample in zip(tree_multi.apply(X_test), X_test):
    pred = leaf_multi_models[leaf_id].predict(sample.reshape(1, -1))[0]
    y_multi_pred.append(pred)
y_multi_pred = np.array(y_multi_pred)

内容的提问来源于stack exchange，提问作者Prabhat