WSL2中TensorFlow GPU环境Conv2D报错：无法加载libcublasLt.so.12

问题场景 #

在WSL2中配置带GPU加速的TensorFlow环境，运行卷积层代码时触发报错，但全连接层代码可正常执行。

报错复现代码 #

>>> from tensorflow import keras
>>> import numpy as np
>>> t = np.ones([5,32,32,3])
>>> c = keras.layers.Conv2D(32, 3, activation="relu")
>>> c(t)

报错信息 #

2023-07-09 09:59:38.820408: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:982] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-07-09 09:59:39.031437: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:982] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-07-09 09:59:39.031864: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:982] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-07-09 09:59:39.034068: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:982] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-07-09 09:59:39.034535: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:982] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-07-09 09:59:39.034921: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:982] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-07-09 09:59:40.590457: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:982] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-07-09 09:59:40.590941: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:982] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-07-09 09:59:40.591052: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1722] Could not identify NUMA node of platform GPU id 0, defaulting to 0.  Your kernel may not have been built with NUMA support.
2023-07-09 09:59:40.591459: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:982] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-07-09 09:59:40.591526: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3858 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5
Could not load library libcublasLt.so.12. Error: libcublasLt.so.12: cannot open shared object file: No such file or directory
Aborted

正常运行的代码（Dense层） #

>>> from tensorflow import keras
>>> import numpy as np
>>> t = np.ones([5,32,32,3])
>>> c = keras.layers.Dense(32, activation="relu")
>>> c(t)

已尝试的无效方案 #

• 重新安装CUDA、CuDNN
• 在全新安装的WSL Ubuntu 20.04和22.04.2中重新配置环境
• 尝试TensorFlow 2.10、2.11、2.12、2.13版本
• 执行apt install libcublasLt安装相关库

环境信息 #

• Windows 11 Home
• WSL 2
• Intel i7-9750h
• NVIDIA RTX 2060 Notebook
• TensorFlow 2.12.1
• Python 3.9
• WSL2 Ubuntu 20.04
• CUDA 11.8
• CuDNN 8.6
• 当前在conda环境中运行

解决方法 #

核心问题是TensorFlow版本与CUDA版本不匹配：TensorFlow 2.12+依赖CUDA 12.x版本的libcublasLt.so.12，而你安装的CUDA 11.8没有该库文件。

方案1：版本适配降级 #

将TensorFlow降级到2.10.x或2.11.x版本，这两个版本官方适配CUDA 11.8，无需改动CUDA环境：

pip install tensorflow==2.11.0

方案2：升级CUDA环境 #

升级CUDA到12.1，并搭配适配的CuDNN 8.9.x：

1. 卸载现有CUDA 11.8，下载并安装CUDA 12.1
2. 下载对应CUDA 12.1的CuDNN 8.9.x，解压后将文件复制到CUDA安装目录
3. 配置环境变量（在~/.bashrc或~/.zshrc中添加）：

   export LD_LIBRARY_PATH=/usr/local/cuda-12.1/lib64:$LD_LIBRARY_PATH
   export PATH=/usr/local/cuda-12.1/bin:$PATH
   

4. 执行source ~/.bashrc生效，验证nvcc -V显示版本为12.1

方案3：conda环境内安装适配库 #

如果不想改动系统级CUDA，可在conda环境内直接安装对应版本的CUDA工具包：

conda install cudatoolkit=12.1 cudnn=8.9 -c conda-forge

关于NUMA警告 #

报错中的NUMA相关提示可忽略，这是WSL2内核默认未开启NUMA支持导致的，不影响GPU加速功能。

内容的提问来源于stack exchange，提问作者Vedant Jumle