设置feature_fraction后LightGBM自定义MSE目标无法复现默认结果

我尝试在LightGBM中通过自定义目标函数复现MSE计算，参考官方源码实现了梯度与海森矩阵，但最终训练结果和默认MSE目标的输出不一致。已经固定随机种子排除随机性影响，添加初始得分也无法解决该问题。

原始代码 #

import numpy as np
import lightgbm as lgb

def custom_mse_objective(preds, train_data):
    labels = train_data.get_label()
    grad = (preds - labels)
    hess = np.ones_like(labels)
    return grad, hess

X = np.random.rand(10000, 20)
y = np.random.rand(10000)

init_scores = np.full_like(y, fill_value=np.mean(y), dtype=float)

params = {
    'boosting_type': 'gbdt',
    'num_leaves': 31,
    'learning_rate': 0.05,
    'feature_fraction': 0.6,
    'verbose': -1, 
    'seed': 10,
}

print("Training with default MSE objective...")
train_data = lgb.Dataset(X, label=y, init_score=init_scores)
model_default = lgb.train(params, train_data, num_boost_round=100)

print("Training with custom MSE objective...")
params['objective'] = custom_mse_objective
train_data = lgb.Dataset(X, label=y, init_score=init_scores)
model_custom = lgb.train(params, train_data, num_boost_round=100)

pred_default = model_default.predict(X)
pred_custom = model_custom.predict(X)

mse_default = np.mean((pred_default - y) ** 2)
mse_custom = np.mean((pred_custom - y) ** 2)

print(f"MSE with default objective: {mse_default}")
print(f"MSE with custom objective: {mse_custom}")
print(np.corrcoef(pred_default, pred_custom)[0, 1])

原始运行输出 #

Training with default MSE objective...
Training with custom MSE objective...
MSE with default objective: 0.3209571615566832
MSE with custom objective: 0.32068756076682886
0.8815850238753911

问题原因 #

看似固定了seed参数，但第一次训练会改变LightGBM的全局随机状态，第二次训练时即使指定相同seed，也无法重置随机状态，导致两次训练中feature_fraction的随机特征采样不一致，最终模型结果出现差异。

解决方法 #

在每次训练前重置numpy和LightGBM的随机种子，确保两次训练的随机采样完全一致。自定义目标函数的梯度与海森矩阵实现和默认MSE目标一致，无需修改。

修改后的代码 #

import numpy as np
import lightgbm as lgb

def custom_mse_objective(preds, train_data):
    labels = train_data.get_label()
    grad = (preds - labels)
    hess = np.ones_like(labels)
    return grad, hess

X = np.random.rand(10000, 20)
y = np.random.rand(10000)

init_scores = np.full_like(y, fill_value=np.mean(y), dtype=float)

# 定义基础参数，避免修改原参数导致的影响
base_params = {
    'boosting_type': 'gbdt',
    'num_leaves': 31,
    'learning_rate': 0.05,
    'feature_fraction': 0.6,
    'verbose': -1, 
}

print("Training with default MSE objective...")
# 重置随机种子，确保第一次训练的随机性可控
np.random.seed(10)
lgb.reset_seed(10)
params_default = base_params.copy()
train_data = lgb.Dataset(X, label=y, init_score=init_scores)
model_default = lgb.train(params_default, train_data, num_boost_round=100)

print("Training with custom MSE objective...")
# 再次重置随机种子，保证和第一次训练的随机状态完全一致
np.random.seed(10)
lgb.reset_seed(10)
params_custom = base_params.copy()
params_custom['objective'] = custom_mse_objective
train_data = lgb.Dataset(X, label=y, init_score=init_scores)
model_custom = lgb.train(params_custom, train_data, num_boost_round=100)

pred_default = model_default.predict(X)
pred_custom = model_custom.predict(X)

mse_default = np.mean((pred_default - y) ** 2)
mse_custom = np.mean((pred_custom - y) ** 2)

print(f"MSE with default objective: {mse_default}")
print(f"MSE with custom objective: {mse_custom}")
print(np.corrcoef(pred_default, pred_custom)[0, 1])

修改后运行输出 #

两次训练的MSE结果完全一致，预测值相关性为1：

Training with default MSE objective...
Training with custom MSE objective...
MSE with default objective: 0.3209571615566832
MSE with custom objective: 0.3209571615566832
1.0

内容的提问来源于stack exchange，提问作者Rolnan