如何在嵌套函数中安全将惰性对象导出至parSapply工作进程?
问题:parSapply运行嵌套函数时适配lm/lmer模型失败
我定义了嵌套函数g与内部函数f,f会处理lm、lme4::lmer或lmerTest::lmer的调用并重新执行eval。f在g中通过vapply(类型1)、mclapply(类型2)或parSapply(类型3)重复执行R次。
代码如下:
g <- \(fit, type=1, R=2L, nc=2L) { f <- \(...) { ## f manipulates call if (inherits(fit, 'lmerMod')) { cl <- fit@call ## manipulate... lme4::fixef(eval(cl, envir=fenv)) } else { cl <- fit$call ## manipulate... eval(cl, envir=fenv)$coe } } fenv <- new.env() environment(f) <- fenv if (type == 1) { ## rep. using `vapply` bc <- vapply(seq_len(R), f, numeric(ncol(model.matrix(fit)))) } else if (type == 2) { ## rep. using `mclapply` bc <- parallel::mclapply(seq_len(R), f, mc.cores=nc) |> do.call(what='cbind') } else if (type == 3) { ## rep. using `parSapply` .CL <- parallel::makeCluster(nc) on.exit(parallel::stopCluster(.CL)) parallel::clusterExport(.CL, c('fit'), envir=fenv) bc <- parallel::parSapply(.CL, seq_len(R), f) } else stop('type undefined') return(bc) }
使用vapply或mclapply时始终正常,但使用parSapply时总是失败,而parSapply正是我当前依赖的方法。同时发现使用lm、lme4::lmer或lmerTest::lmer时报错情况不同,且我需要以对象形式传入公式,这增加了复杂度。
lm案例
fo1 <- mpg ~ cyl g(lm(fo1, mtcars), type=1) ## 正常运行 g(lm(fo1, mtcars), type=2) ## 正常运行 # [,1] [,2] # (Intercept) 37.88458 37.88458 # cyl -2.87579 -2.87579 g(lm(fo1, mtcars), type=3) ## 运行失败 # Error in checkForRemoteErrors(val) : # 2 nodes produced errors; first error: object 'fo1' not found
lme4::lmer/lmerTest::lmer案例,报错情况不同
data('sleepstudy', package='lme4') sleepstudy$foo <- 0 ## to keep promise fo2 <- Reaction ~ Days + (1 | Subject) g(lme4::lmer(fo2, sleepstudy), type=1) ## 正常运行 g(lme4::lmer(fo2, sleepstudy), type=2) ## 正常运行 # [,1] [,2] # (Intercept) 251.40510 251.40510 # Days 10.46729 10.46729 g(lme4::lmer(fo2, sleepstudy), type=3) ## 运行失败 # Error in checkForRemoteErrors(val) : # 2 nodes produced errors; first error: bad 'data': object 'sleepstudy' not # found g(lmerTest::lmer(fo2, sleepstudy), type=1) ## 正常运行 g(lmerTest::lmer(fo2, sleepstudy), type=2) ## 正常运行 g(lmerTest::lmer(fo2, sleepstudy), type=3) ## 运行失败 # Error in checkForRemoteErrors(val) : # 2 nodes produced errors; first error: object 'fo2' not found
lm和lmerTest::lmer均提示找不到公式对象,而lme4::lmer提示找不到sleepstudy数据。
显式指定公式时的parSapply变体
g(lm(mpg ~ cyl, mtcars), type=3) ## 正常运行 # [,1] [,2] # (Intercept) 37.88458 37.88458 # cyl -2.87579 -2.87579 g(lmerTest::lmer(Reaction ~ Days + (1 | Subject), sleepstudy), type=3) ## 正常运行 # [,1] [,2] # (Intercept) 251.40510 251.40510 # Days 10.46729 10.46729 g(lme4::lmer(Reaction ~ Days + (1 | Subject), sleepstudy), type=3) ## 运行失败 # Error in checkForRemoteErrors(val) : # 2 nodes produced errors; first error: bad 'data': object 'sleepstudy' not # found
lme4::lmer仍提示找不到sleepstudy。我尝试调整fenv <- new.env(),去掉它后仅lme4::lmer会像其他模型一样提示找不到公式,其余无变化。
我需要解决如何让parSapply适配lm、lme4::lmer和lmerTest::lmer的问题,怀疑是惰性对象未完整导出至工作进程,也不确定对象所处环境,尝试envir=parent.frame()无效。
> sessionInfo() R version 4.3.1 (2023-06-16) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Ubuntu 22.04.3 LTS Matrix products: default BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.10.0 LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0 locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 [4] LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 [7] LC_PAPER=en_US.UTF-8 LC_NAME=C LC_ADDRESS=C [10] LC_TELEPHONE=C LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C time zone: Europe/Zurich tzcode source: system (glibc) attached base packages: [1] stats graphics grDevices utils datasets methods base loaded via a namespace (and not attached): [1] vctrs_0.6.3 nlme_3.1-163 cli_3.6.1 rlang_1.1.1 [5] generics_0.1.3 minqa_1.2.5 glue_1.6.2 colorspace_2.1-0 [9] lme4_1.1-34 scales_1.2.1 fansi_1.0.4 grid_4.3.1 [13] lmerTest_3.1-3 munsell_0.5.0 tibble_3.2.1 MASS_7.3-60 [17] numDeriv_2016.8-1.1 lifecycle_1.0.3 compiler_4.3.1 dplyr_1.1.2 [21] Rcpp_1.0.11 pkgconfig_2.0.3 rstudioapi_0.15.0 lattice_0.21-8 [25] nloptr_2.0.3 R6_2.5.1 tidyselect_1.2.0 utf8_1.2.3 [29] pillar_1.9.0 parallel_4.3.1 splines_4.3.1 magrittr_2.0.3 [33] Matrix_1.6-1 tools_4.3.1 gtable_0.3.4 matrixStats_1.0.0 [37] boot_1.3-28.1 ggplot2_3.4.3
内容的提问来源于stack exchange,提问作者jay.sf
相关产品推荐
相关产品推荐

