MXNet容器CUDA驱动版本不兼容问题及相关技术疑问

主机环境 #

Linux XYZ 5.15.0-76-generic #83~20.04.1-Ubuntu SMP Wed Jun 21 20:23:31 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

主机NVIDIA驱动与CUDA版本 #

NVIDIA驱动信息 #

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.108.03   Driver Version: 510.108.03   CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| 26%   35C    P8     6W /  75W |    265MiB /  4096MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1063      G   /usr/lib/xorg/Xorg                 92MiB |
|    0   N/A  N/A      1338      G   /usr/bin/gnome-shell               26MiB |
|    0   N/A  N/A      2078      G   /usr/lib/firefox/firefox          144MiB |
+-----------------------------------------------------------------------------+

CUDA版本 #

USER@XYZ:~$ nvcc -V

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Thu_Feb_10_18:23:41_PST_2022
Cuda compilation tools, release 11.6, V11.6.112
Build cuda_11.6.r11.6/compiler.30978841_0

问题详情 #

安装nvidia-container-toolkit以支持容器GPU访问，拉取mxnet/python:1.9.1_gpu_cu112_py3镜像后，执行以下命令启动容器：

docker run -it --runtime=nvidia --gpus all mxnet/python:1.9.1_gpu_cu112_py3 /bin/bash

容器内检查NVIDIA驱动，确认GPU可识别：

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.108.03   Driver Version: 510.108.03   CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| 26%   34C    P8     6W /  75W |    278MiB /  4096MiB |      5%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

容器内显示CUDA版本为11.2，在Python 3.7中执行以下代码验证MXNet GPU支持：

import mxnet as mx
mx.context.num_gpus()

出现如下错误：

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/dist-packages/mxnet/context.py", line 275, in num_gpus
    check_call(_LIB.MXGetGPUCount(ctypes.byref(count)))
  File "/usr/local/lib/python3.7/dist-packages/mxnet/base.py", line 246, in check_call
    raise get_last_ffi_error()
mxnet.base.MXNetError: Traceback (most recent call last):
  File "../include/mxnet/base.h", line 458
CUDA: Check failed: e == cudaSuccess (803 vs. 0) : system has unsupported display driver / cuda driver combination

推测是主机（CUDA 11.6）与容器（CUDA 11.2）版本不兼容导致MXNet无法识别GPU。

临时解决方法 #

删除容器内/usr/local/cuda-11.2/compat/目录下的所有libcuda.so文件及其符号链接后，MXNet可成功识别GPU：

Python 3.7.13 (default, Apr 24 2022, 01:04:09) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import mxnet as mx
>>> mx.context.num_gpus()
1

此时容器内显示CUDA版本与主机一致（11.6）：

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.108.03   Driver Version: 510.108.03   CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| 26%   35C    P8     6W /  75W |    282MiB /  4096MiB |      4%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

疑问 #

看起来nvidia-container-toolkit绑定了主机的libcuda.so，导致主机与容器产生依赖，现提出以下疑问：

• 如何移除这种依赖关系？
• 是否可以通过修改nvidia-container-toolkit的配置文件，避免绑定主机的libcuda.so？
• 该问题是否仅针对MXNet框架？

内容的提问来源于stack exchange，提问作者gunner gunner