OnnxRuntime：仅先导入PyTorch时GPU上的CUDNN推理才生效

已在系统中安装CUDA、CUDNN和onnxruntime-gpu，且确认GPU兼容，但直接启动onnxruntime推理会话时，出现如下警告：

>>> import onnxruntime as rt
>>> rt.get_available_providers()
['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider']
>>> rt.InferenceSession("[PATH TO MODEL .onnx]", providers=['CUDAExecutionProvider'])
2023-01-31 09:07:03.289984495 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:578 CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Please ensure all dependencies are met.
<onnxruntime.capi.onnxruntime_inference_collection.InferenceSession object at 0x7f740b4af100>

但先导入PyTorch再导入onnxruntime，推理可正常在GPU运行，启动会话后nvidia-smi能看到该Python进程：

$ python
Python 3.8.16 (default, Dec  7 2022, 01:12:06)
[GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> import onnxruntime as rt
>>> sess = rt.InferenceSession("PATH TO MODEL .onnx", providers=['CUDAExecutionProvider'])
>>>

导入顺序至关重要：若先导入onnxruntime再导入torch，仍会出现相同警告。

问题根源 #

查看torch包的__init__文件，发现关键操作是全局加载libtorch_global_deps.so库，该操作会解决onnxruntime-gpu依赖的CUDA相关库加载问题。

解决方案 #

方案1：调整导入顺序，先导入PyTorch

确保在导入onnxruntime之前先导入torch：

import torch
import onnxruntime as rt

sess = rt.InferenceSession("PATH TO MODEL .onnx", providers=['CUDAExecutionProvider'])

方案2：手动全局加载libtorch_global_deps.so

无需导入PyTorch，直接通过ctypes加载目标库后再导入onnxruntime：

import ctypes
# 替换为你的虚拟环境中该库的实际路径
lib_path = '[path to my .venv38]/lib/python3.8/site-packages/torch/lib/libtorch_global_deps.so'
ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)

import onnxruntime as rt
sess = rt.InferenceSession("PATH TO MODEL .onnx", providers=['CUDAExecutionProvider'])

已安装版本信息 #

• NVIDIA-SMI 510.108.03
• Driver Version: 510.108.03
• CUDA Version: 11.6
• CuDNN Version: cudnn-11.4-linux-x64-v8.2.4.15
• onnx==1.12.0
• onnxruntime-gpu==1.13.1
• torch==1.12.1+cu116
• torchvision==0.13.1+cu116
• Python版本：3.8
• Ubuntu 22.04 5.19.3-051903-generic

Python包均安装在虚拟环境中。

内容的提问来源于stack exchange，提问作者mutableVoid