mlr3基准与嵌套重采样:从benchmark对象提取调优模型计算特征重要性
问题描述
我在使用mlr3中的benchmark()函数对比多个机器学习算法,其中一个是经过超参数调优的XGB模型。我采用了嵌套重采样方案:外层重采样使用holdout样本评估整体性能,内层重采样使用5折交叉验证完成超参数调优。
除了获取所有机器学习算法的准确率估值外,我还希望查看调优后XGB的特征重要性。要实现这一点需要访问benchmark对象内部的调优模型,但我不知道该如何操作:benchmark()返回的对象是深度嵌套的列表,我不理解它的结构。
已有的公开方案都无法满足我的需求:
- 部分方案使用流水线中的学习器作为示例,和我使用benchmark对象的场景不符
- 部分方案仅展示了如何一次性提取基准测试的所有信息,没有说明如何单独提取基准中某一个学习器的调优后模型
我用于执行嵌套重采样的代码如下,完成基准测试后我需要获取调优后的XGB模型来计算特征重要性:
require(mlr3verse) ### 参数设置 ## 调优相关参数 n_folds = 5 grid_search_resolution = 2 measure = msr("classif.acc") task = tsk("iris") # 开启mlr3调试信息输出 options("mlr3.debug" = TRUE) ### 超参数调优设置 # 内层重采样用的AutoTuner ## 内层重采样设计 inner_resampling = rsmp("cv", folds = n_folds) terminator = trm("none") ## 未调优的XGB学习器 xgb_no_tuning = lrn("classif.xgboost", eval_metric = "mlogloss") set_threads(xgb_no_tuning, n = 6) ## XGB自动调优器配置 xgb_learner_tuning = lrn("classif.xgboost", eval_metric = "mlogloss") xgb_search_space = ps(nrounds = p_int(lower = 100, upper= 500), max_depth = p_int(lower = 3, upper= 10), colsample_bytree = p_dbl(lower = 0.6, upper = 1) ) xgb_tuner = tnr("grid_search", resolution = grid_search_resolution) # 开启隐式并行 set_threads(xgb_learner_tuning, n = 6) xgb_tuned = AutoTuner$new(xgb_learner_tuning, inner_resampling, measure, terminator, xgb_tuner, xgb_search_space, store_tuning_instance = TRUE) ## 外层重采样:holdout outer_resampling = rsmp("holdout") outer_resampling$instantiate(task) bm_design = benchmark_grid( tasks = task, learners = c(lrn("classif.featureless"), xgb_no_tuning, xgb_tuned ), resamplings = outer_resampling ) begin_time = Sys.time() bmr = benchmark(bm_design, store_models = TRUE) duration = Sys.time() - begin_time print(duration) ## 基准测试结果 benchmark_results = bmr$aggregate(measure) print(benchmark_results) ## 概览 mlr3misc::map(as.data.table(bmr)$learner, "model") ## 详细结果 # 输出学习器配置 print(bmr$learners$learner)
解决方案
require(mlr3verse) require(mlr3tuning) require(mlr3misc) ### 参数设置 ## 调优相关参数 n_folds = 5 grid_search_resolution = 2 measure = msr("classif.acc") task = tsk("iris") # 开启mlr3调试信息输出 options("mlr3.debug" = TRUE) ### 超参数调优设置 # 内层重采样用的AutoTuner ## 内层重采样设计 inner_resampling = rsmp("cv", folds = n_folds) terminator = trm("none") ## 未调优的XGB学习器 xgb_no_tuning = lrn("classif.xgboost", eval_metric = "mlogloss") set_threads(xgb_no_tuning, n = 6) ## XGB自动调优器配置 xgb_learner_tuning = lrn("classif.xgboost", eval_metric = "mlogloss") xgb_search_space = ps(nrounds = p_int(lower = 100, upper= 500), max_depth = p_int(lower = 3, upper= 10), colsample_bytree = p_dbl(lower = 0.6, upper = 1) ) xgb_tuner = tnr("grid_search", resolution = grid_search_resolution) # 开启隐式并行 set_threads(xgb_learner_tuning, n = 6) xgb_tuned = AutoTuner$new(xgb_learner_tuning, inner_resampling, measure, terminator, xgb_tuner, xgb_search_space, store_tuning_instance = TRUE) ## 外层重采样:holdout outer_resampling = rsmp("holdout") outer_resampling$instantiate(task) bm_design = benchmark_grid( tasks = task, learners = c(lrn("classif.featureless"), xgb_no_tuning, xgb_tuned ), resamplings = outer_resampling ) begin_time = Sys.time() bmr = benchmark(bm_design, store_models = TRUE) duration = Sys.time() - begin_time print(duration) ## 基准测试结果 benchmark_results = bmr$aggregate(measure) print(benchmark_results) ## 概览 mlr3misc::map(as.data.table(bmr)$learner, "model") ## 详细结果 # 输出学习器配置 print(bmr$learners$learner) ## 特征重要性计算 # 从外层采样结果中提取模型 data = as.data.table(bmr) outer_learners = map(data$learner, "learner") # 本示例中调优XGB是第三个学习器,因此取索引为3的元素 xgb_tuned_model = outer_learners[[3]] print(xgb_tuned_model) # 输出特征重要性(默认计算指标为增益) print(xgb_tuned_model$importance())
内容的提问来源于stack exchange,提问作者AW2
相关产品推荐
相关产品推荐

