已加载libcudart11.2却提示找不到libcudart10.1,是否需降级Cuda?
在Ubuntu 20.04系统中搭建了Python3.7 + CUDA11.2 + 对应版本cuDNN-11.2 + TensorFlow2.5环境,用于训练点云数据集。运行代码时出现异常:系统成功加载libcudart.so.11.0,但提示找不到libcudart.so.10.1,完整错误信息如下:
/home/peach/conda3/envs/wjl/bin/python3.7 /home/peach/pointnet2_python3/part_seg/train.py
2023-07-21 09:06:30.426887: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
Traceback (most recent call last):
File "/home/peach/pointnet2_python3/part_seg/train.py", line 27, in
from log import pointnet2_part_seg
File "/home/peach/pointnet2_python3/part_seg/log/pointnet2_part_seg.py", line 9, in
from pointnet_util import pointnet_sa_module, pointnet_fp_module
File "/home/peach/pointnet2_python3/utils/pointnet_util.py", line 15, in
from tf_sampling import farthest_point_sample, gather_point
File "/home/peach/pointnet2_python3/tf_ops/sampling/tf_sampling.py", line 12, in
sampling_module=tf.load_op_library(os.path.join(BASE_DIR, 'tf_sampling_so.so'))
File "/home/peach/conda3/envs/wjl/lib/python3.7/site-packages/tensorflow/python/framework/load_library.py", line 58, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: libcudart.so.10.1: Unable to open shared object file: there is no file or directory available
提问:是否只能通过降级CUDA来解决该问题?
不需要只能降级CUDA,问题根源是你使用的tf_sampling_so.so是基于CUDA10.1编译的,因此依赖旧版本的CUDA运行时库。以下是几种可行的解决方式:
重新编译自定义OP:
找到tf_ops/sampling目录下的源码文件,修改对应的编译脚本(如tf_sampling_compile.sh),将CUDA路径指定为你的CUDA11.2安装路径(通常是/usr/local/cuda-11.2),同时确保编译时链接TensorFlow2.5的头文件和库。重新编译生成适配CUDA11.2的tf_sampling_so.so,这是最稳妥的长期解决方案。创建软链接临时适配:
如果暂时不想重新编译,可以在CUDA库目录下创建软链接,让系统将libcudart.so.11.0识别为libcudart.so.10.1:sudo ln -s /usr/local/cuda/lib64/libcudart.so.11.0 /usr/local/cuda/lib64/libcudart.so.10.1注意:这种方式存在兼容性风险,若自定义OP依赖CUDA10.1的特定API,可能出现运行错误,仅作为临时应急方案。
安装多版本CUDA并切换环境:
在系统中同时安装CUDA10.1和CUDA11.2,运行PointNet2代码前临时切换环境变量,指向CUDA10.1:export CUDA_HOME=/usr/local/cuda-10.1 export LD_LIBRARY_PATH=/usr/local/cuda-10.1/lib64:$LD_LIBRARY_PATH这种方式可以保留原有CUDA11.2环境,同时满足旧版本OP的依赖需求,适合需要在不同CUDA版本间切换的场景。
内容的提问来源于stack exchange,提问作者peach

