Ubuntu 22.04.03下HPE DL560 Gen11服务器NVIDIA H100驱动无法正常工作求助

各位大佬好，我这边遇到了一个棘手的问题：我的HPE DL560 Gen11服务器上安装了NVIDIA H100显卡，但无论怎么尝试安装驱动，显卡都无法被系统正常识别。下面是我已经试过的方法和遇到的具体错误，恳请大家帮忙分析一下！

已尝试的安装方法及对应问题 #

1. 直接安装NVIDIA官网的.run驱动包 #

安装过程中出现nvidia.ko内核模块加载失败的错误，日志关键内容如下：

ERROR: Unable to load the kernel module 'nvidia.ko'. This happens most frequently when this kernel module was built against the wrong or improperly configured kernel sources, with a version of gcc that differs from the one used to build the target kernel, or if another driver, such as nouveau, is present and prevents the NVIDIA kernel module from obtaining ownership of the NVIDIA device(s), or no NVIDIA device installed in this system is supported by this NVIDIA Linux graphics driver release.

-> Kernel module load error: No such device

内核日志里的核心报错片段：

[65781.378193] VFIO - User Level meta-driver version: 0.3
[65781.561750] nvidia-nvlink: Nvlink Core is being initialized, major device number 510
[65781.563296] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid:
NVRM: BAR0 is 0M @ 0x0 (PCI:0000:1a:00.0)
[65781.563309] nvidia: probe of 0000:1a:00.0 failed with error -1
[65781.563378] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid:
NVRM: BAR0 is 0M @ 0x0 (PCI:0000:49:00.0)
[65781.563395] nvidia: probe of 0000:49:00.0 failed with error -1
[65781.563419] NVRM: The NVIDIA probe routine failed for 2 device(s).
[65781.563420] NVRM: None of the NVIDIA devices were initialized.
[65781.563669] nvidia-nvlink: Unregistered Nvlink Core, major device number 510

2. 通过CUDA 12.3附带驱动安装 #

选择CUDA安装包中包含的驱动一起安装，但遇到了和上面完全一致的nvidia.ko加载错误问题。

3. 使用ubuntu-drivers autoinstall自动安装 #

安装过程没有明显报错，但安装完成后nvidia-smi无法正常工作，显卡在硬件检测中显示为UNCLAIMED状态：

• 执行dpkg -l |grep nvidia可以看到所有535版本的驱动包都已成功安装：

ii  libnvidia-cfg1-535:amd64                   535.129.03-0ubuntu0.22.04.1             amd64        NVIDIA binary OpenGL/GLX configuration library
ii  libnvidia-common-535                       535.129.03-0ubuntu0.22.04.1             all          Shared files used by the NVIDIA libraries
ii  libnvidia-compute-535:amd64                535.129.03-0ubuntu0.22.04.1             amd64        NVIDIA libcompute package
ii  libnvidia-compute-535:i386                 535.129.03-0ubuntu0.22.04.1             i386         NVIDIA libcompute package
ii  libnvidia-decode-535:amd64                 535.129.03-0ubuntu0.22.04.1             amd64        NVIDIA Video Decoding runtime libraries
ii  libnvidia-decode-535:i386                  535.129.03-0ubuntu0.22.04.1             i386         NVIDIA Video Decoding runtime libraries
ii  libnvidia-encode-535:amd64                 535.129.03-0ubuntu0.22.04.1             amd64        NVENC Video Encoding runtime library
ii  libnvidia-encode-535:i386                  535.129.03-0ubuntu0.22.04.1             i386         NVENC Video Encoding runtime library
ii  libnvidia-extra-535:amd64                  535.129.03-0ubuntu0.22.04.1             amd64        Extra libraries for the NVIDIA driver
ii  libnvidia-fbc1-535:amd64                   535.129.03-0ubuntu0.22.04.1             amd64        NVIDIA OpenGL-based Framebuffer Capture runtime library
ii  libnvidia-fbc1-535:i386                    535.129.03-0ubuntu0.22.04.1             i386         NVIDIA OpenGL-based Framebuffer Capture runtime library
ii  libnvidia-gl-535:amd64                     535.129.03-0ubuntu0.22.04.1             amd64        NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii  libnvidia-gl-535:i386                      535.129.03-0ubuntu0.22.04.1             i386         NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii  linux-modules-nvidia-535-6.2.0-39-generic  6.2.0-39.40~22.04.1                     amd64        Linux kernel nvidia modules for version 6.2.0-39
ii  linux-modules-nvidia-535-generic-hwe-22.04 6.2.0-39.40~22.04.1                     amd64        Extra drivers for nvidia-535 for the generic-hwe-22.04 flavour
ii  linux-objects-nvidia-535-6.2.0-39-generic  6.2.0-39.40~22.04.1                     amd64        Linux kernel nvidia modules for version 6.2.0-39 (objects)
ii  linux-signatures-nvidia-6.2.0-39-generic   6.2.0-39.40~22.04.1                     amd64        Linux kernel signatures for nvidia modules for version 6.2.0-39-generic
ii  nvidia-compute-utils-535                   535.129.03-0ubuntu0.22.04.1             amd64        NVIDIA compute utilities
ii  nvidia-driver-535                          535.129.03-0ubuntu0.22.04.1             amd64        NVIDIA driver metapackage
ii  nvidia-firmware-535-535.129.03             535.129.03-0ubuntu0.22.04.1             amd64        Firmware files used by the kernel module
ii  nvidia-kernel-common-535                   535.129.03-0ubuntu0.22.04.1             amd64        Shared files used with the kernel module
ii  nvidia-kernel-source-535                   535.129.03-0ubuntu0.22.04.1             amd64        NVIDIA kernel source package
ii  nvidia-prime                               0.8.17.1                                all          Tools to enable NVIDIA's Prime
ii  nvidia-settings                            510.47.03-0ubuntu1                      amd64        Tool for configuring the NVIDIA graphics driver
ii  nvidia-utils-535                           535.129.03-0ubuntu0.22.04.1             amd64        NVIDIA driver support binaries
ii  screen-resolution-extra                    0.18.2                                  all          Extension for the nvidia-settings control panel
ii  xserver-xorg-video-nvidia-535              535.129.03-0ubuntu0.22.04.1             amd64        NVIDIA binary Xorg driver

• 执行nvidia-smi报错：

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

• 执行lshw -C display显示H100处于未声明状态：

*-display UNCLAIMED
description: 3D controller
product: GH100 [H100 PCIe]
vendor: NVIDIA Corporation
physical id: 0
bus info: pci@0000:1a:00.0
version: a1
width: 64 bits
clock: 33MHz
capabilities: pm msi pciexpress msix cap_list
configuration: latency=0
resources: iomemory:20a00-209ff iomemory:20800-207ff iomemory:20a00-209ff
*-display UNCLAIMED
description: 3D controller
product: GH100 [H100 PCIe]
vendor: NVIDIA Corporation
physical id: 0
bus info: pci@0000:49:00.0
version: a1
width: 64 bits
clock: 33MHz
capabilities: pm msi pciexpress msix cap_list
configuration: latency=0
resources: iomemory:22200-221ff iomemory:22000-21fff iomemory:22200-221ff

额外尝试的排查操作 #

我还试过以下步骤，但都没有解决问题：

• 根据uname -r输出匹配重新安装内核头文件
• 配置nouveau黑名单并更新initramfs
• 完全卸载所有NVIDIA驱动后重新安装

现在实在找不到方向了，各位大佬有什么排查思路或者解决方案吗？任何需要的信息我都可以配合提供！

备注：内容来源于stack exchange，提问作者KTLEE