libcudnn_ops_infer.so.8符号未定义：RabbitMQ触发脚本异常排查

系统及环境信息 #

内存信息 #

MemTotal:       30794980 kB
MemFree:        26650464 kB
MemAvailable:   28247716 kB
Buffers:           73680 kB
Cached:          1840696 kB
SwapCached:            0 kB
Active:           981320 kB
Inactive:        2659044 kB
Active(anon):       1356 kB
Inactive(anon):  1736720 kB
Active(file):     979964 kB
Inactive(file):   922324 kB
Unevictable:       18600 kB
Mlocked:           18600 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                40 kB
Writeback:             0 kB
AnonPages:       1744600 kB
Mapped:           640532 kB
Shmem:              3876 kB
KReclaimable:      92008 kB
Slab:             182620 kB
SReclaimable:      92008 kB
SUnreclaim:        90612 kB
KernelStack:        8256 kB
PageTables:        19732 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    15397488 kB
Committed_AS:    9151572 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       80276 kB
VmallocChunk:          0 kB
Percpu:             6496 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      406328 kB
DirectMap2M:     7979008 kB
DirectMap1G:    25165824 kB

CPU信息 #

Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Byte Order:                         Little Endian
Address sizes:                      46 bits physical, 48 bits virtual
CPU(s):                             8
On-line CPU(s) list:                0-7
Thread(s) per core:                 2
Core(s) per socket:                 4
Socket(s):                          1
NUMA node(s):                       1
Vendor ID:                          GenuineIntel
CPU family:                         6
Model:                              63
Model name:                         Intel(R) Xeon(R) CPU @ 2.30GHz
Stepping:                           0
CPU MHz:                            2299.998
BogoMIPS:                           4599.99
Hypervisor vendor:                  KVM
Virtualization type:                full
L1d cache:                          128 KiB
L1i cache:                          128 KiB
L2 cache:                           1 MiB
L3 cache:                           45 MiB
NUMA node0 CPU(s):                  0-7
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Mitigation; PTE Inversion
Vulnerability Mds:                  Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Meltdown:             Mitigation; PTI
Vulnerability Mmio stale data:      Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Retbleed:             Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI
                                     Syscall hardening, KVM SW loop
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2
                                     ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_kn
                                    own_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c r
                                    drand hypervisor lahf_lm abm invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 avx
                                    2 smep bmi2 erms invpcid xsaveopt arat md_clear arch_capabilities

GPU信息 #

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.171.04             Driver Version: 535.171.04   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   38C    P8               9W /  70W |    105MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+

PyTorch版本信息 #

Name: torch
Version: 2.2.1
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
Home-page: https://pytorch.org/
Author: PyTorch Team
Author-email: packages@pytorch.org
License: BSD-3
Location: /usr/local/lib/python3.8/dist-packages
Requires: nvidia-cufft-cu12, nvidia-cusparse-cu12, nvidia-cuda-cupti-cu12, nvidia-nccl-cu12, nvidia-nvtx-cu12, triton, filelock, fsspec, nvidia-cusolver-cu12, nvidia-cuda-nvrtc-cu12, nvidia-curand-cu12, nvidia-cublas-cu12, sympy, nvidia-cudnn-cu12, jinja2, typing-extensions, nvidia-cuda-runtime-cu12, networkx
Required-by: torchvision

问题背景 #

• 此前脚本因thop库的profile函数报错，已卸载thop模块
• 旧服务器使用Torch 2.2.0（同CUDA 12.2版本）时脚本运行正常，新服务器切换为Torch 2.2.1后出现该报错
• 核心异常：通过RabbitMQ触发脚本时出错，但直接使用相同参数运行脚本完全正常
• 当前使用YOLOv7的attempt_load函数（models.experimental.py）加载模型

排查方向 #

1. 检查环境变量差异：RabbitMQ触发的进程和直接运行的进程可能继承不同的环境变量，重点核对LD_LIBRARY_PATH，确保其优先指向PyTorch自带的cuDNN库目录（如/usr/local/lib/python3.8/dist-packages/nvidia/cudnn/lib），避免系统中其他版本的cuDNN干扰加载。
2. 验证cuDNN版本兼容性：执行pip show nvidia-cudnn-cu12查看当前版本，对比Torch 2.2.1官方要求的依赖版本，若不匹配则卸载后重新安装对应版本。
3. 排查进程启动上下文：确认RabbitMQ worker进程的用户权限、工作目录是否与直接运行时一致，权限不足可能导致无法读取库文件；同时检查虚拟环境是否激活，直接运行时可能手动激活了虚拟环境，但RabbitMQ触发时未自动激活。
4. 重新安装Torch确保完整性：卸载现有Torch及相关CUDA组件：

   pip uninstall -y torch nvidia-cudnn-cu12 nvidia-cuda-runtime-cu12
   

   再用官方命令重新安装适配CUDA 12.2的Torch 2.2.1，确保所有依赖组件完整安装。
5. 适配YOLOv7代码：确认YOLOv7代码是否兼容Torch 2.2.1，检查attempt_load函数中涉及cuDNN初始化的逻辑，可尝试更新YOLOv7到最新版本，或对比旧服务器上的YOLOv7代码是否存在差异。

内容的提问来源于stack exchange，提问作者Malla Raraju