mlr3中ROC曲线提取标准差/95%CI及绘图定制技术问询

我需要提取多模型在任务基准测试中结果的标准差和/或95%置信区间（95%CI），保证结果完整性。之前查过mlr3 k折交叉验证的标准差提取方法，但不确定后续是否有新实现。以下是我的代码、基准测试（bmr）结果及绘图：

代码实现 #

resampling_outer = rsmp("cv", folds = 5)
resampling_inner = rsmp("cv", folds = 3)

set.seed(372)
resampling_outer$instantiate(task_wilcox)
resampling_inner$instantiate(task_wilcox)

at_xgboost = auto_tuner(tuner=tnr("mbo"), learner = xgboost,resampling = resampling_inner, measure = msr("classif.auc"),term_evals = 20,store_tuning_instance = TRUE,store_models = TRUE)
at_ranger = auto_tuner(tuner=tnr("mbo"), learner = ranger,resampling = resampling_inner, measure = msr("classif.auc"),term_evals = 20,store_tuning_instance = TRUE,store_models = TRUE)
at_svm = auto_tuner(tuner=tnr("mbo"), learner = svm,resampling = resampling_inner, measure = msr("classif.auc"),term_evals = 20,store_tuning_instance = TRUE,store_models = TRUE)
at_knn = auto_tuner(tuner=tnr("mbo"), learner = knn,resampling = resampling_inner, measure = msr("classif.auc"),term_evals = 20,store_tuning_instance = TRUE,store_models = TRUE)

learners <- c(at_xgboost, at_svm, at_ranger, at_knn)

measures = msrs(c("classif.auc", "classif.bacc", "classif.bbrier"))

# Benchmarking
set.seed(372)
design = benchmark_grid(tasks = task_wilcox, learners = learners, resamplings = resampling_outer)
bmr = benchmark(design, store_models = TRUE)
results <- bmr$aggregate(measures)
print(results)
autoplot(bmr, measure = msr("classif.auc"))
autoplot(bmr, type = "roc")

绘图结果 #

AUC箱线图 #

ROC曲线 #

输出结果 #

聚合结果（results） #

nr     task_id                learner_id resampling_id iters classif.auc classif.bacc classif.bbrier
1:  1 data_wilcox       scale.xgboost.tuned            cv     5   0.6112939    0.5767294      0.2326787
2:  2 data_wilcox           scale.svm.tuned            cv     5   0.5226407    0.5010260      0.1893202
3:  3 data_wilcox scale.random_forest.tuned            cv     5   0.6200084    0.5614843      0.2229120
4:  4 data_wilcox           scale.knn.tuned            cv     5   0.5731675    0.5002955      0.1917721

内层调优结果 #

extract_inner_tuning_results(bmr)[,list(learner_id, classif.auc)]
                   learner_id classif.auc
 1:       scale.xgboost.tuned   0.6231350
 2:       scale.xgboost.tuned   0.6207103
 3:       scale.xgboost.tuned   0.6175323
 4:       scale.xgboost.tuned   0.6195693
 5:       scale.xgboost.tuned   0.6222398
 6:           scale.svm.tuned   0.5891432
 7:           scale.svm.tuned   0.5837583
 8:           scale.svm.tuned   0.5767444
 9:           scale.svm.tuned   0.6027165
10:           scale.svm.tuned   0.6082825
11: scale.random_forest.tuned   0.6287649
12: scale.random_forest.tuned   0.6165179
13: scale.random_forest.tuned   0.6288599
14: scale.random_forest.tuned   0.6259322
15: scale.random_forest.tuned   0.6234295
16:           scale.knn.tuned   0.5931790
17:           scale.knn.tuned   0.5926835
18:           scale.knn.tuned   0.5931790
19:           scale.knn.tuned   0.5929156
20:           scale.knn.tuned   0.5929156

问题 #

1. ROC曲线上的彩色透明边际是标准差/置信区间，但不知道如何提取这些数据；推测需要从外层重采样结果提取，但暂无直接方法。
2. 每个模型AUC的箱线图构建依据是什么？结果似乎和外层循环（resampling_outer）的测试集结果不符。
3. 如何在mlr3中定制ROC曲线？比如添加AUC值、移除曲线边际等。

1. 提取标准差/95%置信区间数据 #

ROC曲线的边际是外层5折交叉验证中每折ROC曲线的置信区间，要提取这些数据，需按以下步骤操作：

步骤1：获取外层每折的预测结果 #

# 提取所有外层重采样的预测数据
preds = bmr$score()$prediction

# 按学习者分组，拆分每折的预测
preds_list = split(preds, preds$learner_id)

步骤2：计算每折的ROC曲线点 #

library(precrec)

# 定义函数计算单折ROC数据
compute_roc = function(pred) {
  roc_obj = evalmod(scores = pred$prob[,2], labels = pred$truth)
  return(data.frame(
    fpr = roc_obj$curves[[1]]$x,
    tpr = roc_obj$curves[[1]]$y,
    learner = pred$learner_id[1]
  ))
}

# 计算所有折的ROC数据
all_roc_data = lapply(preds_list, function(learner_preds) {
  do.call(rbind, lapply(learner_preds, compute_roc))
})
all_roc_data = do.call(rbind, all_roc_data)

步骤3：计算标准差/95%置信区间 #

library(data.table)
setDT(all_roc_data)

# 按学习者和FPR分组，计算TPR的均值、标准差和95%CI
roc_stats = all_roc_data[, .(
  tpr_mean = mean(tpr),
  tpr_sd = sd(tpr),
  tpr_lower = quantile(tpr, 0.025),
  tpr_upper = quantile(tpr, 0.975)
), by = .(learner, fpr)]

roc_stats中包含了每个学习者在不同FPR下TPR的均值、标准差和95%置信区间，与ROC图里的边际完全对应。

如果要提取外层验证指标（如AUC）的标准差，直接从bmr$score()结果计算即可：

score_data = bmr$score(measures = msr("classif.auc"))
auc_stats = score_data[, .(
  auc_mean = mean(classif.auc),
  auc_sd = sd(classif.auc),
  auc_ci_lower = quantile(classif.auc, 0.025),
  auc_ci_upper = quantile(classif.auc, 0.975)
), by = learner_id]

2. AUC箱线图的构建依据 #

默认的AUC箱线图不是外层交叉验证的测试集结果，而是内层调优过程中产生的性能数据：

• 每个外层折会运行一次内层3折调优（你设置了20次MBO迭代），每次迭代会评估内层3折的AUC均值
• autoplot(bmr, measure = msr("classif.auc"))默认展示的是所有调优迭代的AUC分布，也就是每个学习者对应5个外层折 × 20次迭代 = 100个AUC值

如果想要绘制外层测试集的AUC箱线图，需要手动提取数据绘图：

library(ggplot2)
score_data = bmr$score(msr("classif.auc"))
ggplot(score_data, aes(x = learner_id, y = classif.auc)) +
  geom_boxplot() +
  labs(title = "外层交叉验证AUC箱线图", x = "模型", y = "AUC")

3. 定制ROC曲线 #

mlr3的autoplot基于ggplot2，可通过修改ggplot对象实现各种定制：

移除曲线边际（置信区间） #

直接在autoplot中设置se = FALSE：

autoplot(bmr, type = "roc", se = FALSE)

添加AUC值 #

先计算每个模型的AUC均值，再通过annotate添加标签：

# 计算每个模型的AUC均值
auc_means = bmr$aggregate(msr("classif.auc"))[, .(learner_id, classif.auc)]

# 绘制ROC曲线并添加AUC标签
p = autoplot(bmr, type = "roc")
for (i in 1:nrow(auc_means)) {
  p = p + annotate("text", x = 0.7, y = 0.3 - (i-1)*0.1, 
                   label = paste0(auc_means$learner_id[i], "\nAUC: ", round(auc_means$classif.auc[i], 3)),
                   color = scales::hue_pal()(nrow(auc_means))[i])
}
print(p)

其他定制（主题、颜色、标题等） #

直接使用ggplot2的函数扩展：

p = autoplot(bmr, type = "roc", se = FALSE) +
  ggtitle("定制ROC曲线") +
  xlab("假阳性率(FPR)") +
  ylab("真阳性率(TPR)") +
  theme_minimal() +
  scale_color_brewer(palette = "Set1")
print(p)

内容的提问来源于stack exchange，提问作者NDe