如何在mlr3中用mbo调参器实现贝叶斯超参数优化的可复现性？

我使用R语言的mlr3系列包构建可复现的机器学习流程，尝试将regr.glmboost学习器与mbo调参器、run_time终止器结合进行贝叶斯超参数优化，但在较长运行时间下无法实现结果复现。

复现代码 #

library(mlr3verse)
library(mlr3mbo)
library(mlr3misc)
library(magrittr)
library(nycflights13)

dt <- as.data.table(weather)
dt <- dt[order(time_hour), .(origin = as.factor(origin), month = as.factor(month), hour = as.factor(hour), temp, dewp, humid, wind_dir, wind_speed, precip, visib, pressure, time_hour = as.numeric(time_hour))]
dt <- na.omit(dt)

best_ones <- map_dtr(
  1L:3L,
  function(i) {
    my_learner <- lrn("regr.glmboost",
      family = to_tune(p_fct(levels = c("Gaussian", "Laplace", "Huber"))),
      nuirange = to_tune(p_dbl(lower = 0, upper = 1000, logscale = FALSE)),
      mstop = to_tune(p_int(lower = 1, upper = 3, trafo = function(x) 10**x)),
      nu = to_tune(p_dbl(lower = 0.01, upper = 0.3, logscale = TRUE)),
      risk = to_tune(p_fct(levels = c("inbag", "oobag", "none"))),
      trace = to_tune(c(TRUE, FALSE)),
      stopintern = to_tune(c(TRUE, FALSE))
    )

    my_task <- as_task_regr(
      x = dt,
      target = "pressure",
      id = "weather_data"
    )

    my_instance <- ti(
      task = my_task,
      learner = my_learner,
      resampling = rsmp("cv", folds = 3),
      measure = msr("regr.mae"),
      terminator = trm("run_time", secs = 300)
    )

    my_tuner <- tnr("mbo")

    set.seed(1234L, kind = "L'Ecuyer-CMRG")
    my_tuner$optimize(my_instance)

    my_instance$archive$best()
  }
)

best_ones[]

三次运行的差异结果 #

原因与解决方法 #

核心原因 #

run_time终止器依赖实际运行时长，不同次运行中系统负载、单轮迭代耗时的波动会导致实际完成的迭代次数不一致。即使设置了种子，迭代次数不同也会得到不同的最优结果。此外，种子设置的位置和覆盖范围也需要调整。

解决方案 #

1. 优先使用迭代次数终止器：这是保证复现性的核心，固定迭代次数而非运行时间，确保每次运行的搜索过程完全一致。
2. 统一种子设置：将全局种子设置放在循环外，覆盖所有随机过程；若使用并行计算，L'Ecuyer-CMRG类型的种子能保证并行环境下的复现性。
3. 保留run_time的妥协方案：若必须使用时间终止器，需在mbo控制参数中额外设置种子，但仍可能因系统耗时差异导致复现性不完全。

修改后的可复现代码 #

library(mlr3verse)
library(mlr3mbo)
library(mlr3misc)
library(magrittr)
library(nycflights13)

dt <- as.data.table(weather)
dt <- dt[order(time_hour), .(origin = as.factor(origin), month = as.factor(month), hour = as.factor(hour), temp, dewp, humid, wind_dir, wind_speed, precip, visib, pressure, time_hour = as.numeric(time_hour))]
dt <- na.omit(dt)

# 全局设置可复现种子，覆盖所有随机过程
set.seed(1234L, kind = "L'Ecuyer-CMRG")

best_ones <- map_dtr(
  1L:3L,
  function(i) {
    my_learner <- lrn("regr.glmboost",
      family = to_tune(p_fct(levels = c("Gaussian", "Laplace", "Huber"))),
      nuirange = to_tune(p_dbl(lower = 0, upper = 1000, logscale = FALSE)),
      mstop = to_tune(p_int(lower = 1, upper = 3, trafo = function(x) 10**x)),
      nu = to_tune(p_dbl(lower = 0.01, upper = 0.3, logscale = TRUE)),
      risk = to_tune(p_fct(levels = c("inbag", "oobag", "none"))),
      trace = to_tune(c(TRUE, FALSE)),
      stopintern = to_tune(c(TRUE, FALSE))
    )

    my_task <- as_task_regr(
      x = dt,
      target = "pressure",
      id = "weather_data"
    )

    # 替换为迭代次数终止器，确保每次运行迭代次数一致
    my_instance <- ti(
      task = my_task,
      learner = my_learner,
      resampling = rsmp("cv", folds = 3),
      measure = msr("regr.mae"),
      terminator = trm("evals", n = 20) # 固定迭代次数
    )

    my_tuner <- tnr("mbo")
    my_tuner$optimize(my_instance)

    my_instance$archive$best()
  }
)

best_ones[]

补充说明 #

• 若必须使用run_time终止器，可在初始化mbo调参器时添加控制参数：tnr("mbo", control = mlr3mbo::control_mbo(seed = 1234L))，但仍可能因系统耗时差异导致迭代次数不同，复现性无法完全保证。
• L'Ecuyer-CMRG种子类型适配并行计算场景，后续若启用并行（如future::plan("multisession")），仍能保证结果复现。

内容的提问来源于stack exchange，提问作者sanyi