Conda环境下用Ray做LSTM调参时出现libcudnn_adv_train.so.8加载失败错误

环境信息 #

• 系统：Ubuntu 20.04
• 显卡：Nvidia GTX 1070
• 内存：32GB
• 依赖版本（声明vs实际安装）：

  # 声明版本
  nvidia driver version = 515
  cuda = 11.7
  cudnn = 8.5
  
  # conda实际安装包
  # Name                    Version                   Build  Channel
  cudatoolkit               11.2.2              hbe64b41_10    conda-forge
  cudnn                     8.1.0.77             h90431f1_0    conda-forge
  cudnnenv                  0.8.0                    pypi_0    pypi
  magma-cuda110             2.5.2                         1    pytorch
  

报错信息 #

(BroadModel pid=145826) Could not load library libcudnn_adv_train.so.8. Error: /home/arman/anaconda3/envs/tf/bin/../lib/libcudnn_ops_train.so.8: undefined symbol: _Z22cudnnGenericOpTensorNdILi3EE13cudnnStatus_tP12cudnnContext16cudnnGenericOp_t21cudnnNanPropagation_tPKdPKvPK17cudnnTensorStructS8_S8_SB_S8_S8_SB_Pv, version libcudnn_ops_infer.so.8
(BroadModel pid=145826) *** SIGABRT received at time=1661887480 on cpu 6 ***
(BroadModel pid=145826) PC: @     0x7f0e0703400b  (unknown)  raise
(BroadModel pid=145826)     @     0x7f0e0735b420  (unknown)  (unknown)
(BroadModel pid=145826)     @                0x3  (unknown)  (unknown)
(BroadModel pid=145826) [2022-08-30 15:24:40,346 E 145826 145875] logging.cc:361: *** SIGABRT received at time=1661887480 on cpu 6 ***
(BroadModel pid=145826) [2022-08-30 15:24:40,346 E 145826 145875] logging.cc:361: PC: @     0x7f0e0703400b  (unknown)  raise
(BroadModel pid=145826) [2022-08-30 15:24:40,348 E 145826 145875] logging.cc:361:     @     0x7f0e0735b420  (unknown)  (unknown)
(BroadModel pid=145826) [2022-08-30 15:24:40,351 E 145826 145875] logging.cc:361:     @                0x3  (unknown)  (unknown)
(BroadModel pid=145826) Fatal Python error: Aborted
(BroadModel pid=145826) 
2022-08-30 15:24:40,556 WARNING worker.py:1829 -- A worker died or was killed while 
    executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. 
    RayTask ID: ffffffffffffffffb6fedc6726cbd809885e392701000000 Worker ID: f56690910b7de43cb69060607ad61467ac62cac906e7d85793d2c9a3 Node ID: 7321e7f004a736d6a59ac21c27271e98c304335e7828e7cb458b4599 
    Worker IP address: 10.37.0.16 Worker port: 45455 
    Worker PID: 145826 Worker exit type: SYSTEM_ERROR 
    Worker exit detail: Worker unexpectedly exits with a connection error code 2. 
    End of file. 
    There are some potential root causes. 
    (1) The process is killed by SIGKILL by OOM killer due to high memory usage. 
    (2) ray stop --force is called. 
    (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.

解决方案 #

• 核心问题：版本完全不匹配
  你声明要使用CUDA 11.7+cuDNN 8.5，但conda环境实际安装的是cudatoolkit=11.2.2和cudnn=8.1.0.77，同时TensorFlow 2.9.1官方仅支持CUDA 11.2搭配cuDNN 8.1；若要使用CUDA 11.7，需升级TensorFlow到2.10及以上版本，且对应cuDNN 8.5。

• 统一conda环境依赖

  1. 先卸载冲突包：

     conda remove --force cudatoolkit cudnn magma-cuda110
     pip uninstall -y cudnnenv
     

  2. 二选一执行：
     • 方案1：适配TensorFlow 2.9.1的稳定组合

       conda install -c conda-forge cudatoolkit=11.2.2 cudnn=8.1.0
       

     • 方案2：升级TensorFlow以支持CUDA 11.7

       pip install tensorflow==2.10.0
       conda install -c conda-forge cudatoolkit=11.7 cudnn=8.5.0
       

• 确保环境变量优先级
  避免系统级CUDA库干扰conda环境，运行Ray前执行：

  export LD_LIBRARY_PATH=/home/arman/anaconda3/envs/tf/lib:$LD_LIBRARY_PATH
  

• Ray Worker环境隔离
  Ray启动worker可能继承系统环境，可在Ray初始化时指定环境变量，或在任务函数开头添加：

  import os
  os.environ['LD_LIBRARY_PATH'] = '/home/arman/anaconda3/envs/tf/lib:' + os.environ.get('LD_LIBRARY_PATH', '')
  

内容的提问来源于stack exchange，提问作者Arman Asgharpoor