Windows本地加载HuggingFace量化模型报错求助（Python3.10）

问题描述 #

在Python 3.10.0本地Windows环境中，尝试加载HuggingFace的unsloth/llama-3-8b-Instruct-bnb-4bit模型进行文本摘要时出现报错，但该模型在Google Colab和Kaggle平台可正常运行。

报错信息 #

E:\Internships\ConsciusAI\.venv\lib\site-packages\huggingface_hub\file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
Unused kwargs: ['_load_in_4bit', '_load_in_8bit', 'quant_method']. These kwargs are not used in <class 'transformers.utils.quantization_config.BitsAndBytesConfig'>.
E:\Internships\ConsciusAI\.venv\lib\site-packages\transformers\quantizers\auto.py:159: UserWarning: You passed `quantization_config` or equivalent parameters to `from_pretrained` but the model you're loading already has a `quantization_config` attribute. The `quantization_config` from the model will be used.
  warnings.warn(warning_msg)
Traceback (most recent call last):
  File "E:\Internships\ConsciusAI\email_2.py", line 77, in <module>
    main()
  File "E:\Internships\ConsciusAI\email_2.py", line 71, in main
    summary = summarize_email(content)
  File "E:\Internships\ConsciusAI\email_2.py", line 22, in summarize_email
    pipeline = transformers.pipeline(
  File "E:\Internships\ConsciusAI\.venv\lib\site-packages\transformers\pipelines\__init__.py", line 906, in pipeline
    framework, model = infer_framework_load_model(
  File "E:\Internships\ConsciusAI\.venv\lib\site-packages\transformers\pipelines\base.py", line 283, in infer_framework_load_model
    model = model_class.from_pretrained(model, **kwargs)
  File "E:\Internships\ConsciusAI\.venv\lib\site-packages\transformers\models\auto\auto_factory.py", line 563, in from_pretrained
    return model_class.from_pretrained(
  File "E:\Internships\ConsciusAI\.venv\lib\site-packages\transformers\modeling_utils.py", line 3165, in from_pretrained
    hf_quantizer.validate_environment(
  File "E:\Internships\ConsciusAI\.venv\lib\site-packages\transformers\quantizers\quantizer_bnb_4bit.py", line 62, in validate_environment
    raise ImportError(
ImportError: Using `bitsandbytes` 8-bit quantization requires Accelerate: `pip install accelerate` and the latest version of bitsandbytes: `pip install -i https://pypi.org/simple/ bitsandbytes`

代码示例 #

def summarize_email(content):
    model_id = "unsloth/llama-3-8b-Instruct-bnb-4bit"

    pipeline = transformers.pipeline(
        "text-generation",
        model=model_id,
        model_kwargs={
            "torch_dtype": torch.float16,
            "quantization_config": {"load_in_4bit": True},
            "low_cpu_mem_usage": True,
        },
    )

    messages = [
        {"role": "system", "content": "You are good at Summarizing"},
        {"role": "user", "content": "Summarize the email for me " + content},
    ]

    prompt = pipeline.tokenizer.apply_chat_template(
            messages,
            tokenize=False,
            add_generation_prompt=True
    )

    terminators = [
        pipeline.tokenizer.eos_token_id,
        pipeline.tokenizer.convert_tokens_to_ids("")
    ]

    outputs = pipeline(
        prompt,
        max_new_tokens=256,
        eos_token_id=terminators,
        do_sample=True,
        temperature=0.6,
        top_p=0.9,
    )

解决方案 #

1. 安装/更新必要依赖 #

根据报错提示，执行以下命令安装缺失的accelerate和最新版bitsandbytes：

pip install accelerate
pip install --upgrade bitsandbytes

注：Windows环境下若安装bitsandbytes仍有问题，可尝试安装适配Windows的版本：pip install bitsandbytes-windows

2. 优化代码参数 #

• 移除冗余量化配置：由于unsloth/llama-3-8b-Instruct-bnb-4bit模型本身已内置4bit量化配置，代码中无需再指定"quantization_config": {"load_in_4bit": True}，移除该参数可消除相关警告。
• 修复终止token错误：pipeline.tokenizer.convert_tokens_to_ids("")会触发无效token错误，建议移除该项，仅保留eos_token_id作为终止标记。

修改后的完整代码：

def summarize_email(content):
    model_id = "unsloth/llama-3-8b-Instruct-bnb-4bit"

    pipeline = transformers.pipeline(
        "text-generation",
        model=model_id,
        model_kwargs={
            "torch_dtype": torch.float16,
            "low_cpu_mem_usage": True,
        },
    )

    messages = [
        {"role": "system", "content": "You are good at Summarizing"},
        {"role": "user", "content": "Summarize the email for me " + content},
    ]

    prompt = pipeline.tokenizer.apply_chat_template(
            messages,
            tokenize=False,
            add_generation_prompt=True
    )

    terminators = [pipeline.tokenizer.eos_token_id]

    outputs = pipeline(
        prompt,
        max_new_tokens=256,
        eos_token_id=terminators,
        do_sample=True,
        temperature=0.6,
        top_p=0.9,
    )
    return outputs[0]['generated_text'].split(prompt)[-1].strip()

内容的提问来源于stack exchange，提问作者Aswin Jimmy