使用inputs_embeds调用generate方法报错：LlamaForCausalLM未实现转发

尝试使用inputs_embeds向LlamaForCausalLM模型传递嵌入时触发报错，报错信息：

ValueError: You passed inputs_embeds to .generate(), but the model class LlamaForCausalLM doesn't have its forwarding implemented. See the GPT2 implementation for an example, and feel free to open a PR with it!

嵌入张量形状为torch.Size([1, 46, 4096])，相关代码如下：

if True:
    from unsloth import FastLanguageModel
    model, tokenizer = FastLanguageModel.from_pretrained(
        model_name = "/content/drive/My Drive/finetuneunslothllama", 
        max_seq_length = max_seq_length,
        dtype = dtype,
        load_in_4bit = load_in_4bit,
    )
    FastLanguageModel.for_inference(model) # Activer native 2x faster inference

outputs = model.generate(inputs_embeds=embeddings, max_new_tokens=64, use_cache=True)

generated_text = tokenizer.batch_decode(outputs, skip_special_tokens=True)
print(generated_text)

LlamaForCausalLM的generate方法原生未实现inputs_embeds参数的转发逻辑，模型生成阶段无法直接接收预计算的嵌入张量作为输入，只能通过常规的input_ids路径处理输入。

提供两种可行解决思路：

思路一：转换为input_ids输入（最简便） #

如果嵌入张量是tokenizer生成的，可直接用tokenizer获取input_ids替代；若是自定义嵌入，可通过相似度匹配映射到对应token id：

import torch
with torch.no_grad():
    # 获取模型词嵌入矩阵
    embed_matrix = model.get_input_embeddings().weight
    # 计算每个嵌入向量与矩阵的相似度，取最匹配的token id
    similarities = torch.matmul(embeddings, embed_matrix.T)
    input_ids = similarities.argmax(dim=-1)

# 用input_ids调用generate
outputs = model.generate(input_ids=input_ids, max_new_tokens=64, use_cache=True)

思路二：修改模型类以支持inputs_embeds #

参考GPT2的实现逻辑，给LlamaForCausalLM添加inputs_embeds处理逻辑，可通过补丁方式修改模型的generate方法：

from transformers.models.llama.modeling_llama import LlamaForCausalLM

def patched_generate(self, inputs_embeds=None, **kwargs):
    if inputs_embeds is not None:
        batch_size, seq_len, _ = inputs_embeds.shape
        device = inputs_embeds.device
        # 生成必要的attention_mask和position_ids
        attention_mask = torch.ones((batch_size, seq_len), device=device)
        position_ids = torch.arange(seq_len, device=device).unsqueeze(0).repeat(batch_size, 1)
        
        # 调用模型forward获取初始状态
        outputs = self.forward(
            inputs_embeds=inputs_embeds,
            attention_mask=attention_mask,
            position_ids=position_ids,
            use_cache=True,
            return_dict=True
        )
        # 将初始状态传入原生generate逻辑
        kwargs["past_key_values"] = outputs.past_key_values
        kwargs["inputs_embeds"] = None
        kwargs["position_ids"] = position_ids[:, -1:] + 1
        return super().generate(**kwargs)
    else:
        return super().generate(**kwargs)

# 给模型实例打补丁
LlamaForCausalLM.generate = patched_generate

# 现在可正常调用
outputs = model.generate(inputs_embeds=embeddings, max_new_tokens=64, use_cache=True)

注意：若使用Unsloth加速，需在FastLanguageModel.for_inference(model)执行后再打补丁，确保补丁作用到加速后的模型实例。

内容的提问来源于stack exchange，提问作者Steven Thorn