Llama-3微调模型部署SageMaker遇.half()量化模型不支持错误求助

问题描述 #

微调Llama-3 8B模型后，使用DeepSpeed镜像在AWS SageMaker部署时触发以下错误：

.half() is not supported for quantized model. Please use the model as it is, since the model has already been casted to the correct dtype.

错误栈核心逻辑：

deepspeed.inference.engine.py -> _convert_to_dtype -> model.half() -> transformers.modeling_utils.py -> ValueError

根因：模型已通过BitsAndBytesConfig加载为4bit量化模型，DeepSpeed初始化时指定dtype=model.dtype（即float16），导致DeepSpeed尝试对量化模型执行.half()类型转换，而量化模型不支持该操作。

解决方案 #

1. 移除DeepSpeed初始化中的dtype参数 #

修改get_model函数内的deepspeed.init_inference调用，去掉dtype=model.dtype参数，让DeepSpeed自动适配模型的现有 dtype，避免强制转换：

修改后的get_model函数片段：

def get_model(properties):
    tensor_parallel = properties["tensor_parallel_degree"]
    local_rank = int(os.getenv("LOCAL_RANK", "0"))

    model, tokenizer = init_model(model_name="cs_model")
    # 移除dtype参数，避免DeepSpeed强制转换量化模型类型
    model = deepspeed.init_inference(
        model,
        mp_size=tensor_parallel,
        replace_method="auto",
        replace_with_kernel_inject=True,
    )
    generator = pipeline(
        task="text-generation", model=model, tokenizer=tokenizer, device=local_rank
    )
    return generator

2. 验证模型加载配置 #

确保init_model中的量化配置正确，模型加载后保持量化状态，无需额外类型转换：

• load_in_4bit=True已启用4bit量化
• bnb_4bit_compute_dtype=torch.float16仅用于计算时的 dtype，模型本身仍为4bit量化状态

3. 确认DeepSpeed兼容性 #

使用的DeepSpeed版本（0.12.6）支持4bit量化模型的推理加速，replace_method="auto"会自动适配模型结构进行内核注入优化。

修改后的完整代码 #

from djl_python import Input, Output
import os
import deepspeed
import torch
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    TrainingArguments,
    pipeline,
    logging,
)

predictor = None

def init_model(model_name="cs_model"):
    compute_dtype = getattr(torch, "float16")
    bnb_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=compute_dtype,
        bnb_4bit_use_double_quant=False,
    )
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        device_map="auto",
        quantization_config=bnb_config,
        trust_remote_code=True,
    )
    model.config.use_cache = False
    model.config.pretraining_tp = 1
    tokenizer = AutoTokenizer.from_pretrained(
        model_name,
        trust_remote_code=True,
    )
    tokenizer.pad_token = tokenizer.eos_token
    tokenizer.padding_side = "right"
    return model, tokenizer

def get_model(properties):
    tensor_parallel = properties["tensor_parallel_degree"]
    local_rank = int(os.getenv("LOCAL_RANK", "0"))

    model, tokenizer = init_model(model_name="cs_model")
    model = deepspeed.init_inference(
        model,
        mp_size=tensor_parallel,
        replace_method="auto",
        replace_with_kernel_inject=True,
    )
    generator = pipeline(
        task="text-generation", model=model, tokenizer=tokenizer, device=local_rank
    )
    return generator


def handle(inputs: Input) -> None:
    global predictor
    if not predictor:
        predictor = get_model(inputs.get_properties())

    if inputs.is_empty():
        # Model server makes an empty call to warmup the model on startup
        return None

    data = inputs.get_as_string()
    result = predictor(data, do_sample=True, max_new_tokens=256)
    return Output().add(result)

内容的提问来源于stack exchange，提问作者M80