CUDA12+tf-nightly2.12无法检测GPU，PyTorch可正常使用求助

• tf-nightly版本 = 2.12.0-dev2023203
• Python版本 = 3.10.6
• CUDA驱动版本 = 525.85.12
• CUDA版本 = 12.0
• Cudnn版本 = 8.5.0
• 系统：Linux（x86_64，Ubuntu 22.04）
• 开发环境：Visual Studio Code，基于venv虚拟环境

使用tf-nightly 2.12（适配CUDA 12.0）在NVIDIA GeForce RTX 3050 GPU上运行模型时，GPU无法被检测。所有CUDA相关检查结果正常，且PyTorch可正常调用该GPU，以下是检查步骤及报错信息，求解决方案。

1. 检查CUDA版本
   执行命令：

$ nvcc --version

输出：

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Fri_Jan__6_16:45:21_PST_2023
Cuda compilation tools, release 12.0, V12.0.140
Build cuda_12.0.r12.0/compiler.32267302_0

2. 检查CUDA库连接
   执行命令：

$ echo $LD_LIBRARY_PATH

输出：

/usr/cuda/lib

3. 检查GPU驱动及虚拟环境访问权限
   执行命令：

$ nvidia-smi

输出：

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC || Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. ||                               |                      |               MIG M. ||===============================+======================+======================||   0  NVIDIA GeForce ...  On   | 00000000:01:00.0  On |                  N/A || N/A   40C    P5     6W /  20W |     46MiB /  4096MiB |     22%      Default ||                               |                      |                  N/A |+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                                  ||  GPU   GI   CI        PID   Type   Process name                  GPU Memory ||        ID   ID                                                   Usage      ||=============================================================================||    0   N/A  N/A      1356      G   /usr/lib/xorg/Xorg                 45MiB |+-----------------------------------------------------------------------------+

4. 添加并检查CUDA/bin路径
   执行命令：

$ export PATH="/usr/local/cuda/bin:$PATH"

再执行：

$ echo $PATH

输出：

/usr/local/cuda-12.0/bin:/home/victus-linux/Escritorio/MasterThesis_CODE/to_share/venv_master/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin

5. 检查CUDA安装（自定义函数）
   执行脚本：

function lib_installed() { /sbin/ldconfig -N -v $(sed 's/:/ /' <<< $LD_LIBRARY_PATH) 2>/dev/null | grep $1; }
function check() { lib_installed $1 && echo "$1 is installed" || echo "ERROR: $1 is NOT installed"; }
check libcuda
check libcudart

输出：

libcudart.so.12 -> libcudart.so.12.0.146
        libcuda.so.1 -> libcuda.so.525.85.12
        libcuda.so.1 -> libcuda.so.525.85.12
        libcudadebugger.so.1 -> libcudadebugger.so.525.85.12
libcuda is installed
        libcudart.so.12 -> libcudart.so.12.0.146
libcudart is installed

6. 检查Cudnn安装（自定义函数）
   执行脚本：

function lib_installed() { /sbin/ldconfig -N -v $(sed 's/:/ /' <<< $LD_LIBRARY_PATH) 2>/dev/null | grep $1; }
function check() { lib_installed $1 && echo "$1 is installed" || echo "ERROR: $1 is NOT installed"; }
check libcudnn 

输出：

libcudnn_cnn_train.so.8 -> libcudnn_cnn_train.so.8.8.0
        libcudnn_cnn_infer.so.8 -> libcudnn_cnn_infer.so.8.8.0
        libcudnn_adv_train.so.8 -> libcudnn_adv_train.so.8.8.0
        libcudnn.so.8 -> libcudnn.so.8.8.0
        libcudnn_ops_train.so.8 -> libcudnn_ops_train.so.8.8.0
        libcudnn_adv_infer.so.8 -> libcudnn_adv_infer.so.8.8.0
        libcudnn_ops_infer.so.8 -> libcudnn_ops_infer.so.8.8.0
libcudnn is installed

执行检测脚本：

import tensorflow as tf

print(f'\nTensorflow version = {tf.__version__}\n')
print(f'\n{tf.config.list_physical_devices("GPU")}\n')

输出：

2023-03-02 12:05:09.463343: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-03-02 12:05:09.489911: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-03-02 12:05:09.490522: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-03-02 12:05:10.066759: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT

Tensorflow version = 2.12.0-dev20230203

2023-03-02 12:05:10.748675: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-03-02 12:05:10.771263: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1956] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...

[]

执行脚本：

import torch

print(f'\nAvailable cuda = {torch.cuda.is_available()}')

print(f'\nGPUs availables = {torch.cuda.device_count()}')

print(f'\nCurrent device = {torch.cuda.current_device()}')

print(f'\nCurrent Device location = {torch.cuda.device(0)}')

print(f'\nName of the device = {torch.cuda.get_device_name(0)}')

输出：

Available cuda = True

GPUs availables = 1

Current device = 0

Current Device location = <torch.cuda.device object at 0x7fbe26fd2ec0>

Name of the device = NVIDIA GeForce RTX 3050 Laptop GPU

1. 修正LD_LIBRARY_PATH路径
   当前LD_LIBRARY_PATH指向/usr/cuda/lib，但CUDA 12.0的标准库路径为/usr/local/cuda-12.0/lib64，执行以下命令更新环境变量：

export LD_LIBRARY_PATH="/usr/local/cuda-12.0/lib64:$LD_LIBRARY_PATH"

可将该命令添加到虚拟环境的activate脚本中，确保每次激活环境时自动生效。

2. 升级TensorFlow版本
   tf-nightly 2.12-dev20230203对CUDA 12.0的支持可能不完善，建议更新至更高版本的tf-nightly，或切换到正式版TensorFlow 2.13+（正式支持CUDA 12.0）。

3. 验证TensorFlow依赖库
   在虚拟环境中执行以下命令，查看TensorFlow依赖的CUDA库是否可被找到：

ldd $(python -c "import tensorflow as tf; print(tf.__file__)")

若存在缺失库，可手动创建软链接或调整库路径。

4. 升级CuDNN版本
   TensorFlow 2.12+搭配CUDA 12.0时，推荐使用CuDNN 8.6.0及以上版本，尝试升级CuDNN至匹配版本。

内容的提问来源于stack exchange，提问作者JaimeCorton