R在doParallel的foreach循环中调用keras::unserialize_model()出现段错误崩溃

背景 #

我并非R开发者，正在将他人编写的R代码迁移至生产运行环境。本地单独执行模型反序列化代码时完全正常，但将逻辑放入doParallel的foreach循环中执行时，会触发段错误崩溃。

本地正常执行的代码 #

# 本地反序列化模型
my_model1 <- keras::unserialize_model(ser_model1)
my_model2 <- keras::unserialize_model(ser_model2)
my_model3 <- keras::unserialize_model(ser_model3)
my_model4 <- keras::unserialize_model(ser_model4)
my_model5 <- keras::unserialize_model(ser_model5)

并行执行时的崩溃情况 #

将上述反序列化逻辑放入doParallel的foreach循环后，调用keras::unserialize_model(ser_model1)时直接触发段错误崩溃，报错信息如下：

2024-04-06 21:32:07.768352: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-04-06 21:32:07.773084: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-04-06 21:32:07.834125: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-06 21:32:09.108273: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT

*** caught segfault ***
address (nil), cause 'memory not mapped'

Traceback:
1: conditionMessage_from_py_exception(c)
2: conditionMessage.python.builtin.BaseException(errorValue)
3: conditionMessage(errorValue)
4: sprintf("task %d failed - \"%s\"", errorIndex, conditionMessage(errorValue))
5: e$fun(obj, substitute(ex), parent.frame(), e$data)
6: Redacted foreach statement
7: calling_my_function_above()
8: perform_model(inputs)
An irrecoverable exception occurred. R is aborting now ...
Segmentation fault (core dumped)

已尝试的操作 #

• 将线程数从8调整为2，问题依旧存在
• 仅注释掉my_model1 <- keras::unserialize_model(ser_model1)这一行后，代码可正常运行

疑问 #

1. TensorRT警告是否与此崩溃问题相关？
2. 如何排查ser_model1导致崩溃的原因？
3. 为何调用栈中的“task failed”信息未打印？
4. 在涉及多库依赖的情况下，如何调试R的段错误问题？

会话信息 #

R version 4.3.3 (2024-02-29)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.1 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.10.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

time zone: UTC
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base

other attached packages:
[1] dplyr_1.1.4  rjson_0.2.21 hash_2.2.6.3 DBI_1.2.2    odbc_1.4.2

loaded via a namespace (and not attached):
 [1] utf8_1.2.4       R6_2.5.1         tidyselect_1.2.1 bit_4.0.5
 [5] magrittr_2.0.3   glue_1.7.0       bspm_0.5.5.1     blob_1.2.4
 [9] tibble_3.2.1     pkgconfig_2.0.3  generics_0.1.3   bit64_4.0.5
[13] lifecycle_1.0.4  cli_3.6.2        fansi_1.0.6      vctrs_0.6.5
 hms_1.1.3        pillar_1.9.0     Rcpp_1.0.12
[21] rlang_1.1.3

内容的提问来源于stack exchange，提问作者MikeB