加载Google Deplot模型触发ImportError：Torch与CUDA版本适配问题

我尝试使用以下代码从Hugging Face Transformers加载视觉Transformer模型Google Deplot：

model = Pix2StructForConditionalGeneration.from_pretrained('google/deplot')
processor = Pix2StructProcessor.from_pretrained('google/deplot')

但出现如下错误，当前环境配置为Torch2.0.1、Torchvision0.15.2、CUDA=12.0：

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In[17], line 1
----> 1 model = Pix2StructForConditionalGeneration.from_pretrained('google/deplot')
      2 processor = Pix2StructProcessor.from_pretrained('google/deplot')
      3 url = "https://raw.githubusercontent.com/vis-nlp/ChartQA/main/ChartQA%20Dataset/val/png/5090.png"


ImportError: /usr/local/lib/python3.8/dist-packages/fused_layer_norm_cuda.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN3c106detail14torchCheckFailEPKcS2_jRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

环境依赖列表：

Package                      Version
---------------------------- --------------------------
cuda-python                  12.1.0rc1+1.g9e30ea2.dirty
cudf                         22.12.0
cugraph                      22.12.0
cugraph-dgl                  22.12.0
cugraph-service-client       22.12.0
cugraph-service-server       22.12.0
cuml                         22.12.0
cupy-cuda12x                 12.0.0a2
numpy                        1.22.2
nvidia-cublas-cu11           11.10.3.66
nvidia-cuda-cupti-cu11       11.7.101
nvidia-cuda-nvrtc-cu11       11.7.99
nvidia-cuda-runtime-cu11     11.7.99
nvidia-cudnn-cu11            8.5.0.96
nvidia-cufft-cu11            10.9.0.58
nvidia-curand-cu11           10.2.10.91
nvidia-cusolver-cu11         11.4.0.1
nvidia-cusparse-cu11         11.7.4.91
nvidia-dali-cuda110          1.22.0
nvidia-nccl-cu11             2.14.3
nvidia-nvtx-cu11             11.7.91
nvidia-pyindex               1.0.9
openai                       0.27.8
opencv                       4.6.0
python-hostlist              1.23.0
pytorch-lightning            1.2.1
pytorch-quantization         2.1.2
sentence-transformers        2.2.2
torch                        2.0.1
torch-tensorrt               1.4.0.dev0
torchaudio                   2.0.2
torchinfo                    1.8.0
torchtext                    0.13.0a0+fae8e8c
torchvision                  0.15.2

解决思路 #

问题分析 #

这个错误是fused_layer_norm_cuda.so无法找到PyTorch的核心符号，根源是CUDA版本与PyTorch及相关依赖库版本不兼容：

• 系统CUDA版本为12.0，但环境中安装了大量cu11系列的NVIDIA依赖库（如nvidia-cuda-runtime-cu11）
• PyTorch 2.0.1默认安装包基于CUDA 11.7编译，和CUDA 12.0及cu12系列库存在冲突

具体解决步骤 #

1. 统一CUDA与PyTorch版本
   重新安装适配CUDA 12.0的PyTorch包：

   pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
   

2. 卸载冲突的CUDA 11.x依赖库
   删除环境中与CUDA 12.0冲突的cu11系列库：

   pip uninstall -y nvidia-cuda-runtime-cu11 nvidia-cudnn-cu11 nvidia-cublas-cu11 nvidia-cuda-nvrtc-cu11
   

3. 更新Transformers相关依赖
   确保Transformers及加速库适配当前PyTorch版本：

   pip install --upgrade transformers accelerate
   

4. 验证环境一致性
   运行以下代码确认CUDA环境匹配：

   import torch
   print(torch.cuda.is_available())
   print(torch.version.cuda)
   

   需保证输出的CUDA版本与系统CUDA 12.0一致。

内容的提问来源于stack exchange，提问作者Robin Lee