使用bitsandbytes量化Mistral-7B时Import Error的解决方法

问题背景 #

在Hugging Face Spaces搭建基于Mistral-7B-v0.1的Gradio聊天机器人，因模型体积过大必须量化，使用bitsandbytes做4bit量化时出现ImportError。本地树莓派运行相同代码也触发相同错误，排除平台差异问题。

报错信息 #

The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
Traceback (most recent call last):
File "/home/user/app/app.py", line 15, in
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1", quantization_config=quantization_config, device_map="auto", token=access_token)
File "/usr/local/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 563, in from_pretrained
return model_class.from_pretrained(
File "/usr/local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3165, in from_pretrained
hf_*********.validate_environment(
File "/usr/local/lib/python3.10/site-packages/transformers/quantizers/quantizer_bnb_4bit.py", line 62, in validate_environment
raise ImportError(
ImportError: Using bitsandbytes 8-bit quantization requires Accelerate: pip install accelerate and the latest version of bitsandbytes: pip install -i https://pypi.org/simple/ bitsandbytes

环境与已尝试操作 #

• 运行环境：16GB内存的免费CPU，Torch无GPU编译支持
• 已在requirements.txt中添加accelerate和bitsandbytes，也试过指定bitsandbytes==0.43.1，问题未解决

完整代码（app.py） #

import os
import bitsandbytes as bnb
import torch
import gradio as gr
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

access_token = os.environ["GATED_ACCESS_TOKEN"]

quantization_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype="float16",
)

model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1", quantization_config=quantization_config, device_map="auto", token=access_token)
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")

# Function to generate text using the model
def generate_text(prompt):
    text = prompt
    inputs = tokenizer(text, return_tensors="pt")
    
    outputs = model.generate(**inputs, max_new_tokens=20)
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

# Create the Gradio interface
iface = gr.Interface(
    fn=generate_text,
    inputs=[
        gr.inputs.Textbox(lines=5, label="Input Prompt"),
    ],
    outputs=gr.outputs.Textbox(label="Generated Text"),
    title="MisTRALText Generation",
    description="Use this interface to generate text using the MisTRAL language model.",
)

# Launch the Gradio interface
iface.launch()

解决方案 #

方案1：改用transformers原生CPU量化 #

bitsandbytes的4bit/8bit量化仅支持GPU，CPU环境可直接使用transformers的int8量化功能，16GB内存足以承载：

修改代码中的模型加载部分：

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.1",
    device_map="cpu",
    load_in_8bit=True,
    torch_dtype=torch.int8,
    token=access_token
)

需确保transformers版本≥4.35.0，accelerate版本≥0.24.0。

方案2：使用gguf预量化模型（适配CPU/ARM设备） #

Mistral-7B有社区预量化的gguf格式模型，通过llama-cpp-python加载，内存占用更低，适合树莓派等ARM设备：

1. 更新requirements.txt：

gradio
llama-cpp-python>=0.2.50
transformers

2. 修改app.py代码：

import os
import gradio as gr
from llama_cpp import Llama

# 加载4bit量化的gguf模型，需提前将模型文件上传至项目目录
llm = Llama(
    model_path="mistral-7b-v0.1.Q4_K_M.gguf",
    n_ctx=2048,
    n_threads=4,  # 根据CPU核心数调整
    n_gpu_layers=0  # CPU环境设为0
)

def generate_text(prompt):
    output = llm(
        prompt=prompt,
        max_tokens=20,
        stop=["</s>"],
        echo=False
    )
    return output["choices"][0]["text"]

iface = gr.Interface(
    fn=generate_text,
    inputs=gr.Textbox(lines=5, label="Input Prompt"),
    outputs=gr.Textbox(label="Generated Text"),
    title="MisTRALText Generation",
    description="Use this interface to generate text using the MisTRAL language model."
)

iface.launch()

方案3：放弃bitsandbytes，使用GPTQ量化（CPU兼容） #

部分GPTQ量化模型支持CPU推理，可直接加载Hugging Hub上的GPTQ预量化版本：

model = AutoModelForCausalLM.from_pretrained(
    "TheBloke/Mistral-7B-v0.1-GPTQ",
    device_map="cpu",
    token=access_token,
    revision="gptq-4bit-32g-actorder_True"
)

需安装auto-gptq库，在requirements.txt中添加auto-gptq==0.7.1。

内容的提问来源于stack exchange，提问作者Anish