Windows11下RTX A6000用bitsandbytes调用Falcon7B时CUDA初始化失败

环境信息 #

• 操作系统：Windows 11
• GPU：NVIDIA RTX A6000
• CUDA版本：12.1（nvcc --version输出）

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Wed_Feb__8_05:53:42_Coordinated_Universal_Time_2023
Cuda compilation tools, release 12.1, V12.1.66
Build cuda_12.1.r12.1/compiler.32415258_0

问题描述 #

运行调用Falcon7B模型的代码时，频繁触发RuntimeError: CUDA Setup failed despite GPU being available错误，仅偶尔能正常执行。此前排查过程：

• 未安装CUDA时首次出现该错误；
• 安装CUDA 12.1后短暂恢复正常，次日错误复现；
• 将Python版本降级至3.11.1以下后问题解决，但今日错误再次出现。

代码导入部分 #

import torch
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, TrainingArguments, GenerationConfig
from peft import LoraConfig, get_peft_model, PeftConfig, PeftModel, prepare_model_for_kbit_training
from trl import SFTTrainer
import warnings
warnings.filterwarnings("ignore")

报错信息 #

RuntimeError: 
        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues



RuntimeError: Failed to import transformers.training_args because of the following error (look up to see its traceback):

        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

排查与解决步骤 #

1. 匹配bitsandbytes与CUDA版本
   bitsandbytes对Windows平台的CUDA 12.x支持存在版本适配要求，安装对应CUDA 12.1的预编译版本：

   pip install bitsandbytes==0.41.1.post2 --index-url https://download.pytorch.org/whl/cu121
   

2. 确认系统环境变量配置
   检查CUDA的bin目录是否添加到系统PATH，路径通常为C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\bin。打开命令提示符输入nvcc --version，确保能正常输出版本信息。

3. 创建独立虚拟环境避免依赖冲突
   降级Python后复现问题多因依赖包更新冲突，建议搭建干净的虚拟环境：

   conda create -n falcon_env python=3.10
   conda activate falcon_env
   pip install torch==2.0.1+cu121 torchvision==0.15.2+cu121 torchaudio==2.0.2+cu121 --index-url https://download.pytorch.org/whl/cu121
   pip install transformers datasets peft trl bitsandbytes==0.41.1.post2
   

4. 强制指定CUDA路径
   在代码开头添加环境变量配置，强制bitsandbytes加载指定CUDA库：

   import os
   os.environ['BITSANDBYTES_NOWELCOME'] = '1'
   os.environ['CUDA_PATH'] = 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.1'
   

5. 验证bitsandbytes状态
   运行python -m bitsandbytes查看CUDA库加载详情，若提示DLL缺失，可手动从CUDA安装目录复制对应文件到Python环境的Lib/site-packages/bitsandbytes目录下。

内容的提问来源于stack exchange，提问作者Aditi Raghunandan