Optuna与scikit-learn二分类任务AUC ROC结果不一致问题

我正在开展一项二分类任务，利用酶底物的约30个特征预测EC1和EC2，采用XGBoost结合Optuna进行超参数调优，但发现Optuna输出的AUC ROC值与scikit-learn计算的结果存在明显差异：

Optuna输出的最优AUC结果 #

AUC ROC score 1: 0.7109184689577985
AUC ROC score 2: 0.6030927230046949

用最优参数重新训练后scikit-learn计算的AUC结果 #

AUC ROC score 1: 0.7065598459411416
AUC ROC score 2: 0.5656470070422535

实现代码 #

import xgboost as xgb
import optuna
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
import numpy as np

# Setting a fixed random seed for reproducibility
np.random.seed(42)

def train_model(x_train, y_train, x_eval, y_eval):
    def objective(trial):
        param = {
            'objective': 'binary:logistic',
            'eval_metric': 'auc',
            'n_estimators': trial.suggest_int('n_estimators', 100, 1000),
            'max_depth': trial.suggest_int('max_depth', 3, 6),
            'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.1, log=True),
            'subsample': trial.suggest_float('subsample', 0.5, 1),
            'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1),
            'reg_alpha': trial.suggest_float('reg_alpha', 0, 10),
            'reg_lambda': trial.suggest_float('reg_lambda', 0, 10),
            'gamma': trial.suggest_float('gamma', 0.01, 1, log=True),
            'random_state': 42,
            'early_stopping_rounds': 10
        }

        model = xgb.XGBClassifier(**param)

        model.fit(x_train, y_train, eval_set=[(x_eval, y_eval)], verbose=False)
        y_pred = model.predict_proba(x_eval)[:, 1]
        auc_roc = roc_auc_score(y_eval, y_pred)

        return auc_roc

    study = optuna.create_study(direction='maximize')
    study.optimize(objective, n_trials=100)

    return study.best_trial.params, study.best_trial.value

# Splitting the data into train and evaluation sets
x_train, x_eval, y_train, y_eval = train_test_split(x_train, y_train, test_size=0.2, random_state=42)

# For EC1
best_params_1, best_auc_1 = train_model(x_train, y_train[:, 0], x_eval, y_eval[:, 0])
classifier_1 = xgb.XGBClassifier(**best_params_1)
classifier_1.fit(x_train, y_train[:, 0])
y_pred_1 = classifier_1.predict_proba(x_eval)[:, 1]

# For EC2
best_params_2, best_auc_2 = train_model(x_train, y_train[:, 1], x_eval, y_eval[:, 1])
classifier_2 = xgb.XGBClassifier(**best_params_2)
classifier_2.fit(x_train, y_train[:, 1])
y_pred_2 = classifier_2.predict_proba(x_eval)[:, 1]

auc_score_1 = roc_auc_score(y_eval[:, 0], y_pred_1)
auc_score_2 = roc_auc_score(y_eval[:, 1], y_pred_2)

原因分析 #

• 早停策略的差异：Optuna的objective函数中，模型训练时启用了early_stopping_rounds=10并传入eval_set，训练会在验证集性能不再提升时提前停止，实际训练的树数量远小于n_estimators设定值。但后续重新训练时未设置early_stopping_rounds和eval_set，模型会训练完所有n_estimators棵树，导致最终模型与Optuna中评估的最优模型结构不同，AUC结果自然存在差异。
• 模型迭代次数不一致：Optuna返回的best_trial.value是基于早停后最优迭代次数（model.best_iteration_）的模型计算的，而重新训练时用的是原始n_estimators，两者的模型复杂度和拟合效果不同。

解决方法 #

方法1：重新训练时保留早停逻辑 #

复刻Optuna中的训练流程，在后续训练最优模型时加入eval_set和early_stopping_rounds参数：

# 修改后的EC1训练代码
best_params_1, best_auc_1 = train_model(x_train, y_train[:, 0], x_eval, y_eval[:, 0])
classifier_1 = xgb.XGBClassifier(**best_params_1)
# 加入eval_set和early_stopping_rounds，与Optuna内训练逻辑一致
classifier_1.fit(x_train, y_train[:, 0], eval_set=[(x_eval, y_eval)], verbose=False)
y_pred_1 = classifier_1.predict_proba(x_eval)[:, 1]

# EC2同理
best_params_2, best_auc_2 = train_model(x_train, y_train[:, 1], x_eval, y_eval[:, 1])
classifier_2 = xgb.XGBClassifier(**best_params_2)
classifier_2.fit(x_train, y_train[:, 1], eval_set=[(x_eval, y_eval)], verbose=False)
y_pred_2 = classifier_2.predict_proba(x_eval)[:, 1]

方法2：提取Optuna中最优模型的迭代次数 #

修改train_model函数，返回最优参数时将n_estimators替换为早停后的最优迭代次数：

def train_model(x_train, y_train, x_eval, y_eval):
    def objective(trial):
        param = {
            'objective': 'binary:logistic',
            'eval_metric': 'auc',
            'n_estimators': trial.suggest_int('n_estimators', 100, 1000),
            'max_depth': trial.suggest_int('max_depth', 3, 6),
            'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.1, log=True),
            'subsample': trial.suggest_float('subsample', 0.5, 1),
            'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1),
            'reg_alpha': trial.suggest_float('reg_alpha', 0, 10),
            'reg_lambda': trial.suggest_float('reg_lambda', 0, 10),
            'gamma': trial.suggest_float('gamma', 0.01, 1, log=True),
            'random_state': 42,
            'early_stopping_rounds': 10
        }

        model = xgb.XGBClassifier(**param)
        model.fit(x_train, y_train, eval_set=[(x_eval, y_eval)], verbose=False)
        y_pred = model.predict_proba(x_eval)[:, 1]
        auc_roc = roc_auc_score(y_eval, y_pred)
        # 存储最优迭代次数
        trial.set_user_attr("best_iteration", model.best_iteration_)
        return auc_roc

    study = optuna.create_study(direction='maximize')
    study.optimize(objective, n_trials=100)

    best_params = study.best_trial.params
    # 更新n_estimators为最优迭代次数
    best_params['n_estimators'] = study.best_trial.user_attrs["best_iteration"]
    return best_params, study.best_trial.value

# 后续训练直接使用更新后的参数，无需早停
best_params_1, best_auc_1 = train_model(x_train, y_train[:, 0], x_eval, y_eval[:, 0])
classifier_1 = xgb.XGBClassifier(**best_params_1)
classifier_1.fit(x_train, y_train[:, 0])
y_pred_1 = classifier_1.predict_proba(x_eval)[:, 1]

两种方法都能让重新训练的模型与Optuna中评估的最优模型保持一致，消除AUC结果的差异。

内容的提问来源于stack exchange，提问作者Tanmay Gejapati