R并行计算触发"Ancestor must be an environment"错误的原因是什么
问题背景
我使用foreach开展并行计算同时处理大量时间序列,在compute_slope()函数中执行如下操作:
lBd <- floor(TMax^delta) # 下界 uBd <- ceiling(m * TMax^delta) # 上界 # process是包含`n`和`variance`两列的tibble process %>% dplyr::filter(between(n, lBd, uBd)) %>% lm(data = ., log(variance) ~ log(n)) %>% coefficients() %>% .[2]
逻辑很简单:通过参数TMax、delta和m计算截断上下界,用filter()对时间序列做左右截断后,在截断数据上运行线性回归取斜率系数。
大部分时候代码运行正常,但偶发报错,更长的时间序列(即TMax更大)更易触发问题,但触发规律并不固定,报错信息如下:
✖ Problem with `filter()` input `..1`. ℹ Input `..1` is `between(n, lBd, uBd)`. ✖ `ancestor` must be an environment"
我完全无法理解该报错的含义,也没能复现该"ancestor"报错。我尝试了如下测试场景:
library(tidyverse) # 常规场景,可正常运行 mpg %>% filter(between(hwy, 30, 31)) #> # A tibble: 11 x 11 #> manufacturer model displ year cyl trans drv cty hwy fl class #> <chr> <chr> <dbl> <int> <int> <chr> <chr> <int> <int> <chr> <chr> #> 1 audi a4 2 2008 4 manual~ f 20 31 p comp~ #> 2 audi a4 2 2008 4 auto(a~ f 21 30 p comp~ #> 3 chevrolet malibu 2.4 2008 4 auto(l~ f 22 30 r mids~ #> 4 hyundai sonata 2.4 2008 4 auto(l~ f 21 30 r mids~ #> 5 hyundai sonata 2.4 2008 4 manual~ f 21 31 r mids~ #> 6 nissan altima 2.5 2008 4 auto(a~ f 23 31 r mids~ #> 7 toyota camry 2.4 2008 4 manual~ f 21 31 r mids~ #> 8 toyota camry 2.4 2008 4 auto(l~ f 21 31 r mids~ #> 9 toyota camry s~ 2.4 2008 4 manual~ f 21 31 r comp~ #> 10 toyota camry s~ 2.4 2008 4 auto(s~ f 22 31 r comp~ #> 11 toyota corolla 1.8 1999 4 auto(l~ f 24 30 r comp~ # 边界变量未定义 mpg %>% filter(between(hwy, x, 31)) #> Error: Problem with `filter()` input `..1`. #> i Input `..1` is `between(hwy, x, 31)`. #> x object 'x' not found # 边界为函数类型 mpg %>% filter(between(hwy, slice, 31)) #> Error: Problem with `filter()` input `..1`. #> i Input `..1` is `between(hwy, slice, 31)`. #> x cannot coerce type 'closure' to vector of type 'double'
上述测试均返回可解释的正常报错,和我遇到的ancestor报错不同。我怀疑报错与并行处理环节的异常有关,希望获得该ancestor报错的触发示例,以便反向定位代码问题。
问题更新
我在脚本中添加回溯后仍无法定位并行环节的问题,回溯信息如下:
Error in { : task 34 failed - "Problem with `mutate()` column `grid_estimates`. ℹ `grid_estimates = map(data, ~estimate_var_on_grid(process = ., TMax = TMax, grid = grid))`. ✖ Problem with `mutate()` column `slope`. ℹ `slope = map2_dbl(m, delta, ~compute_slope(process, .x, .y, TMax))`. ✖ could not find function "::"" Calls: compute_metrics_on_stable_splits ... tibble -> tibble_quos -> eval_tidy -> %dopar% -> <Anonymous> 11: (function () traceback(2))() 10: stop(simpleError(msg, call = expr)) 9: e$fun(obj, substitute(ex), parent.frame(), e$data) 8: foreach(i = itx, .packages = c("tidyverse", "yardstick", "rsample"), .export = #vector of exports removed for legibility ) %dopar% { i %>% pull(splits) %>% .[[1]] %>% train_and_test(., train_grid = grid, my_mset = my_mset, method = method, TMax = TMax_eval) } } 7: eval_tidy(xs[[j]], mask) 6: tibble_quos(xs, .rows, .name_repair) 5: tibble(metrics = .) 4: list2(...) 3: bind_cols(select(splits, alpha), .) 2: foreach(i = itx, .packages = c("tidyverse", "yardstick", "rsample"), .export = #vector of exports removed for legibility ) %dopar% { i %>% pull(splits) %>% .[[1]] %>% train_and_test(., train_grid = grid, my_mset = my_mset, method = method, TMax = TMax_eval) } } %>% tibble(metrics = .) %>% bind_cols(select(splits, alpha), .) 1: compute_metrics_on_stable_splits(method = method, grid = grid, my_mset = metric_set(accuracy, mcc, sens, spec), TMax_eval = TMax_eval, v = 40)
此时报错变为could not find function "::",和之前的ancestor报错一样无法解释。我还曾经遇到过如下报错:
'rho' must be an environment not pairlist: detected in C-level eval
脚本代码未做任何修改的情况下,报错类型会发生变化,甚至有时可以完全运行成功。目前没有任何排查思路,希望获得相关建议。
会话信息
R version 4.1.2 (2021-11-01) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Red Hat Enterprise Linux 8.2 (Ootpa) Matrix products: default BLAS/LAPACK: /pfs/data5/software_uc2/all/toolkit/Intel_OneAPI/mkl/2021.4.0/lib/intel64/libmkl_intel_lp64.so.1 locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 [7] LC_PAPER=en_US.UTF-8 LC_NAME=C [9] LC_ADDRESS=C LC_TELEPHONE=C [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C attached base packages: [1] parallel stats graphics grDevices utils datasets methods [8] base other attached packages: [1] yardstick_0.0.9 doParallel_1.0.16 iterators_1.0.13 foreach_1.5.1 [5] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.7 purrr_0.3.4 [9] readr_2.1.1 tidyr_1.1.4 tibble_3.1.6 ggplot2_3.3.5 [13] tidyverse_1.3.1 loaded via a namespace (and not attached): [1] tidyselect_1.1.1 haven_2.4.3 colorspace_2.0-2 vctrs_0.3.8 [5] generics_0.1.1 utf8_1.2.2 rlang_0.4.12 pillar_1.6.4 [9] glue_1.5.1 withr_2.4.3 DBI_1.1.1 dbplyr_2.1.1 [13] modelr_0.1.8 readxl_1.3.1 lifecycle_1.0.1 plyr_1.8.6 [17] munsell_0.5.0 gtable_0.3.0 cellranger_1.1.0 rvest_1.0.2 [21] codetools_0.2-18 tzdb_0.2.0 fansi_0.5.0 broom_0.7.10 [25] Rcpp_1.0.7 scales_1.1.1 backports_1.4.0 jsonlite_1.7.2 [29] fs_1.5.1 hms_1.1.1 stringi_1.7.6 grid_4.1.2 [33] cli_3.1.0 tools_4.1.2 magrittr_2.0.1 crayon_1.4.2 [37] pkgconfig_2.0.3 ellipsis_0.3.2 xml2_1.3.3 pROC_1.18.0 [41] reprex_2.0.1 lubridate_1.8.0 assertthat_0.2.1 httr_1.4.2 [45] rstudioapi_0.13 R6_2.5.1 compiler_4.1.2
排查与解决方案
这类偶发、无规律、报错指向底层环境/基础函数缺失的问题,本质都是foreach并行时工作节点的环境损坏或污染导致的,对应你使用的doParallel+foreach组合,可按以下优先级排查:
- 规范并行集群的初始化与销毁逻辑
每次运行并行任务前,先显式销毁旧集群再新建集群,避免重复运行脚本时残留的异常节点被复用:
# 初始化前清理旧集群 if (exists("cl")) { parallel::stopCluster(cl) rm(cl) } # 新建集群并注册 cl <- parallel::makeCluster(parallel::detectCores() - 1) doParallel::registerDoParallel(cl)
任务运行结束后也要显式关闭集群:
parallel::stopCluster(cl) rm(cl)
- 规避tidyverse非标准评估(NSE)的环境查找风险
你当前使用的dplyr::filter(between(n, lBd, uBd))依赖rlang的环境查找逻辑,并行节点环境异常时很容易出现环境指针错误(你遇到的ancestor/rho报错本质都是环境类型错误)。可修改代码为显式传入参数,或者用基础R函数替代tidyverse的NSE操作:
# 修改compute_slope内的过滤逻辑,避免非标准评估 compute_slope <- function(process, m, delta, TMax) { lBd <- floor(TMax^delta) uBd <- ceiling(m * TMax^delta) # 用基础R逻辑替换dplyr filter process_trunc <- process[process$n >= lBd & process$n <= uBd, ] lm(log(variance) ~ log(n), data = process_trunc)$coefficients[[2]] }
- 补全并行导出参数
确保所有并行块内部调用的自定义函数(包括compute_slope、estimate_var_on_grid、train_and_test)都被显式加入.export列表,不要依赖foreach的自动导出逻辑,自动导出经常出现环境绑定异常。另外可把base包加入.packages列表,避免出现::函数找不到的问题:
foreach(i = itx, .packages = c("base", "tidyverse", "yardstick", "rsample"), .export = c("compute_slope", "estimate_var_on_grid", "train_and_test") ) %dopar% { # 任务逻辑 }
- 更换更稳定的并行框架
如果修改后还是偶发报错,可以替换为furrr+future的并行框架,环境处理比foreach+doParallel更稳定,对tidyverse的兼容性也更好。
补充ancestor报错的触发逻辑:当rlang执行非标准评估时,查找的父环境指针被错误指向了非环境类型的对象(比如并行节点环境被意外回收/覆盖为列表/数值)就会触发该报错,属于底层环境异常,和业务逻辑无关。
内容的提问来源于stack exchange,提问作者AlbertRapp

