使用--gpu all启动Docker容器失败，提示libnvidia-ml.so.1找不到的问题求助

问题描述 #

我尝试用docker run --gpu all启动容器时，遇到了如下错误：

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'

nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown.

ERRO[0001] error waiting for container:

已确认正常的情况 #

之前我用conda运行torch也遇到过问题，后来重新安装了CUDA和torch后恢复正常，所以我猜测Docker的问题应该和GPU驱动无关。

我尝试过的解决方案 #

• 参考相关问题尝试开启GPU持久化模式，执行nvidia-smi -pm 1后发现持久化模式本来就是开启状态：

  Persistence mode is already Enabled for GPU 00000000:01:00.0.
  All done.
  

• 尝试编辑/lib/systemd/system/nvidia-persistenced.service文件，修改后没有任何效果
• 按照NVIDIA文档尝试手动启动持久化服务，执行sudo -i后运行nvidia-persistenced --user h.bonnavaud，提示初始化失败；查看服务状态发现它已经正常运行：

  ● nvidia-persistenced.service - NVIDIA Persistence Daemon
  Loaded: loaded (/lib/systemd/system/nvidia-persistenced.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/nvidia-persistenced.service.d
  └─override.conf
  Active: active (running) since Wed 2023-09-27 14:30:25 CEST; 7min ago
  Process: 12354 ExecStart=/usr/bin/nvidia-persistenced --user nvidia-persistenced --persistence-mode --verbose (code=exited, status=0/SUCCESS)
  Main PID: 12355 (nvidia-persiste)
  Tasks: 1 (limit: 38097)
  Memory: 304.0K
  CGroup: /system.slice/nvidia-persistenced.service
  └─12355 /usr/bin/nvidia-persistenced --user nvidia-persistenced --persistence-mode --verbose
  
  Sep 27 14:30:25 PORT-BONNAV-l systemd[1]: Starting NVIDIA Persistence Daemon...
  Sep 27 14:30:25 PORT-BONNAV-l nvidia-persistenced[12355]: Verbose syslog connection opened
  Sep 27 14:30:25 PORT-BONNAV-l nvidia-persistenced[12355]: Now running with user ID 129 and group ID 137
  Sep 27 14:30:25 PORT-BONNAV-l nvidia-persistenced[12355]: Started (12355)
  Sep 27 14:30:25 PORT-BONNAV-l nvidia-persistenced[12355]: device 0000:01:00.0 - registered
  Sep 27 14:30:25 PORT-BONNAV-l nvidia-persistenced[12355]: device 0000:01:00.0 - persistence mode enabled.
  Sep 27 14:30:25 PORT-BONNAV-l nvidia-persistenced[12355]: device 0000:01:00.0 - NUMA memory onlined.
  Sep 27 14:30:25 PORT-BONNAV-l nvidia-persistenced[12355]: Local RPC services initialized
  Sep 27 14:30:25 PORT-BONNAV-l systemd[1]: Started NVIDIA Persistence Daemon.
  

当前状态总结 #

• 排除GPU驱动问题：容器外可以通过CUDA正常使用GPU（torch运行正常）
• 排除持久化模式问题：nvidia-persistenced服务已正常运行，GPU持久化模式已开启
• 不带--gpu all参数时，Docker容器可以正常启动
• 容器外GPU功能正常

系统详情 #

• 操作系统：Ubuntu 20.04

sudo lshw -C display输出 #

~$ sudo lshw -C display
*-display
description: 3D controller
product: NVIDIA Corporation
vendor: NVIDIA Corporation
physical id: 0
bus info: pci@0000:01:00.0
logical name: /dev/fb0
version: a1
width: 64 bits
clock: 33MHz
capabilities: pm msi pciexpress bus_master cap_list rom fb
configuration: depth=32 driver=nvidia latency=0 mode=1920x1080 visual=truecolor xres=1920 yres=1080
resources: iomemory:600-5ff iomemory:600-5ff irq:185 memory:a3000000-a3ffffff memory:6000000000-600fffffff memory:6010000000-6011ffffff ioport:3000(size=128)
*-display
description: VGA compatible controller
product: Intel Corporation
vendor: Intel Corporation
physical id: 2
bus info: pci@0000:00:02.0
version: 01
width: 64 bits
clock: 33MHz
capabilities: pciexpress msi pm vga_controller bus_master cap_list rom
configuration: driver=i915 latency=0
resources: iomemory:600-5ff iomemory:400-3ff irq:184 memory:6072000000-6072ffffff memory:4000000000-400fffffff ioport:4000(size=64) memory:c0000-dffff memory:4010000000-4016ffffff memory:4020000000-40ffffffff

nvidia-smi输出 #

~$ nvidia-smi
Wed Sep 27 14:53:26 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.43.04    Driver Version: 515.43.04    CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA T600 Lap...  On   | 00000000:01:00.0 Off |                  N/A |
| N/A   54C    P8    N/A /  N/A |    603MiB /  4096MiB |      6%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      2498      G   /usr/lib/xorg/Xorg                 86MiB |
|    0   N/A  N/A      3590      G   /usr/lib/xorg/Xorg                203MiB |
|    0   N/A  N/A      3785      G   /usr/bin/gnome-shell               92MiB |
|    0   N/A  N/A      5163      G   /usr/lib/firefox/firefox          145MiB |
|    0   N/A  N/A      7384      G   ...b/thunderbird/thunderbird       67MiB |
+-----------------------------------------------------------------------------+

sudo lshw -c video | grep 'configuration'输出 #

~$ sudo lshw -c video | grep 'configuration'
configuration: depth=32 driver=nvidia latency=0 mode=1920x1080 visual=truecolor xres=1920 yres=1080
configuration: driver=i915 latency=0

希望各位能给我一些解决思路或者提示，谢谢！

备注：内容来源于stack exchange，提问作者Hedwin Bonnavaud