运行meta-llama/Llama-2-7b-chat-hf时遇ImportError问题求助

问题重现 #

运行Llama-2-7b-chat-hf的代码如下：

import torch
from llama_index.llms.huggingface import HuggingFaceLLM

llm = HuggingFaceLLM(
    context_window=4096,
    max_new_tokens=256,
    generate_kwargs={"temperature": 0.0, "do_sample": False},
    system_prompt = "You need to create proposal documents for the information that we are giving to you. This proposal has to be 5 paragraphs and answered well. If you need more information regarding the topic, you can reply with what furhter information could help",
    query_wrapper_prompt = "<|USER|>{query_str}<|ASSISTANT|>",
    tokenizer_name="meta-llama/Llama-2-7b-chat-hf",
    model_name="meta-llama/Llama-2-7b-chat-hf",
    device_map="auto",
    model_kwargs={"torch_dtype": torch.float16 , "load_in_8bit":True}
)

报错信息：

ImportError                               Traceback (most recent call last)
<ipython-input-34-0d2d206e16bb> in <cell line: 4>()
      2 from llama_index.llms.huggingface import HuggingFaceLLM
      3 
----> 4 llm = HuggingFaceLLM(
      5     context_window=4096,
      6     max_new_tokens=256,

3 frames
/usr/local/lib/python3.10/dist-packages/transformers/quantizers/quantizer_bnb_8bit.py in validate_environment(self, *args, **kwargs)
     60     def validate_environment(self, *args, **kwargs):
     61         if not (is_accelerate_available() and is_bitsandbytes_available()):
---&gt; 62             raise ImportError(
     63                 "Using `bitsandbytes` 8-bit quantization requires Accelerate: `pip install accelerate` "
     64                 "and the latest version of bitsandbytes: `pip install -i https://pypi.org/simple/ bitsandbytes`"

ImportError: Using `bitsandbytes` 8-bit quantization requires Accelerate: `pip install accelerate` and the latest version of bitsandbytes: `pip install -i https://pypi.org/simple/ bitsandbytes`

解决方案 #

1. 彻底重装依赖库 #

先卸载现有版本避免冲突：

pip uninstall -y accelerate bitsandbytes

安装最新兼容版本：

pip install --upgrade accelerate bitsandbytes

云环境（如Colab）可指定适配版本：

pip install bitsandbytes==0.41.1.post1

2. 验证依赖导入情况 #

运行代码确认库安装成功：

import accelerate
import bitsandbytes
print("accelerate版本:", accelerate.__version__)
print("bitsandbytes版本:", bitsandbytes.__version__)

无报错后再重新运行LLM初始化代码。

3. 检查CUDA环境兼容性 #

8-bit量化依赖CUDA，确认GPU加速可用：

import torch
print("CUDA可用:", torch.cuda.is_available())
print("CUDA版本:", torch.version.cuda)

若CUDA不可用，禁用8-bit量化并修改参数：

model_kwargs={"torch_dtype": torch.float32}

4. 备选方案：直接禁用8-bit量化 #

若上述方法无效，关闭8-bit量化（需足够显存）：

llm = HuggingFaceLLM(
    context_window=4096,
    max_new_tokens=256,
    generate_kwargs={"temperature": 0.0, "do_sample": False},
    system_prompt = "You need to create proposal documents for the information that we are giving to you. This proposal has to be 5 paragraphs and answered well. If you need more information regarding the topic, you can reply with what furhter information could help",
    query_wrapper_prompt = "<|USER|>{query_str}<|ASSISTANT|>",
    tokenizer_name="meta-llama/Llama-2-7b-chat-hf",
    model_name="meta-llama/Llama-2-7b-chat-hf",
    device_map="auto",
    model_kwargs={"torch_dtype": torch.float16}
)

内容的提问来源于stack exchange，提问作者Aryan Gupta