tidymodels中计算性能指标置信区间报错及替代方法咨询

问题描述 #

运行代码mars_boot <- int_pctl(mars_res, alpha = 0.10)时触发错误：

mars_boot <- int_pctl(mars_res, alpha = 0.10)
Error in UseMethod("int_pctl") : 
  no applicable method for 'int_pctl' applied to an object of class "c('tune_results', 'tbl_df', 'tbl', 'data.frame')"

已知int_pctl仅支持bootstraps()创建的bootstrap重采样对象，当前mars_res是tune_results类对象，使用的R环境信息如下：

sessionInfo()
R version 4.3.2 (2023-10-31 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 11 x64 (build 22631)

Matrix products: default


locale:
[1] LC_COLLATE=Chinese (Simplified)_China.utf8 
[2] LC_CTYPE=Chinese (Simplified)_China.utf8   
[3] LC_MONETARY=Chinese (Simplified)_China.utf8
[4] LC_NUMERIC=C                                
[5] LC_TIME=Chinese (Simplified)_China.utf8    

time zone: Asia/Shanghai
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods  
[7] base      

other attached packages:
 [1] earth_5.3.2        plotmo_3.6.2       TeachingDemos_2.12
 [4] plotrix_3.8-4      Formula_1.2-5      yardstick_1.3.1   
 [7] workflowsets_1.0.1 workflows_1.1.4    tune_1.1.2        
[10] tidyr_1.3.0        tibble_3.2.1       rsample_1.2.1     
[13] recipes_1.0.9      purrr_1.0.2        parsnip_1.2.1     
[16] modeldata_1.3.0    infer_1.0.5        ggplot2_3.5.1     
[19] dplyr_1.1.4        dials_1.2.0        scales_1.3.0      
[22] broom_1.0.5        tidymodels_1.1.1  

loaded via a namespace (and not attached):
 [1] gtable_0.3.4        lattice_0.21-9      vctrs_0.6.5        
 [4] tools_4.3.2         generics_0.1.3      parallel_4.3.2     
 [7] fansi_1.0.6         pkgconfig_2.0.3     Matrix_1.6-1.1     
[10] data.table_1.15.2   lhs_1.1.6           GPfit_1.0-8        
[13] lifecycle_1.0.4     compiler_4.3.2      munsell_0.5.0      
[16] codetools_0.2-19    DiceDesign_1.10     class_7.3-22       
[19] prodlim_2023.08.28  modelenv_0.1.1      pillar_1.9.0       
[22] furrr_0.3.1         MASS_7.3-60         gower_1.0.1        
[25] iterators_1.0.14    rpart_4.1.21        foreach_1.5.2      
[28] parallelly_1.36.0   lava_1.7.3          tidyselect_1.2.0   
[31] digest_0.6.33       future_1.33.0       listenv_0.9.0      
[34] splines_4.3.2       grid_4.3.2          colorspace_2.1-0   
[37] cli_3.6.2           magrittr_2.0.3      survival_3.5-7     
[40] utf8_1.2.4          future.apply_1.11.0 withr_2.5.2        
[43] backports_1.4.1     lubridate_1.9.3     timechange_0.2.0   
[46] globals_0.16.2      nnet_7.3-19         timeDate_4032.109  
[49] hardhat_1.3.0       rlang_1.1.2         Rcpp_1.0.11        
[52] glue_1.6.2          ipred_0.9-14        rstudioapi_0.15.0  
[55] R6_2.5.1   

解决方案 #

1. 适配tune_results对象使用int_pctl #

先从调优结果中提取最优模型，再基于原始数据集生成bootstrap重采样对象，即可使用int_pctl：

# 提取最优模型参数并生成最终模型
best_mars <- select_best(mars_res)
final_mars <- finalize_model(
  mars_spec, # 替换为你的原始模型spec对象
  best_mars
)

# 对原始数据集做bootstrap重采样
set.seed(123)
boot_samples <- bootstraps(your_data, times = 1000) # your_data替换为你的原始数据集

# 在每个bootstrap样本上拟合模型并计算性能指标
boot_results <- boot_samples %>%
  mutate(
    model = map(splits, ~fit(final_mars, data = analysis(.x))),
    metrics = map(model, ~metrics(.x, data = assessment(.x), truth = your_truth_col, estimate = your_estimate_col)) # 替换为你的真实值列和预测值列
  )

# 计算90%置信区间
mars_boot <- int_pctl(boot_results, alpha = 0.10)

2. 其他简便计算方法 #

方法一：基于现有重采样结果计算分位数置信区间

如果模型调优时已经使用了交叉验证等重采样策略，可直接提取所有重采样的性能指标计算置信区间：

# 提取所有重采样的性能指标（不做汇总）
resampled_metrics <- collect_metrics(mars_res, summarize = FALSE)

# 计算90%置信区间
ci <- resampled_metrics %>%
  group_by(.metric) %>%
  summarize(
    lower = quantile(.estimate, 0.05),
    upper = quantile(.estimate, 0.95),
    mean_estimate = mean(.estimate)
  )

方法二：使用infer包快速生成bootstrap置信区间

利用infer包的语法可以更简洁地完成计算：

# 基于最优模型生成预测值，再做bootstrap置信区间计算
boot_ci <- your_data %>%
  mutate(pred = predict(final_mars, new_data = .)) %>%
  specify(response = your_truth_col, prediction = pred) %>%
  calculate(stat = "your_metric") # 替换为你的目标性能指标，如"accuracy"或"rmse"
  generate(reps = 1000, type = "bootstrap") %>%
  get_confidence_interval(level = 0.90)

内容的提问来源于stack exchange，提问作者ayue