使用llama-cpp-python调用Holo1-7B模型无法正确识别图像求助

我是该领域新手，尝试使用llama-cpp-python调用Holo1-7B.i1-Q5_K_M.gguf多模态模型处理图像，但模型输出的图像描述完全不符合实际内容。以下是我的代码、输出结果：

代码 #

from llama_cpp import Llama
import base64

llm = Llama(
    model_path='Holo1-7B.i1-Q5_K_M.gguf',
    n_gpu_layers=-1,
)

def image_to_base64_data_uri(file_path):
    with open(file_path, "rb") as img_file:
        base64_data = base64.b64encode(img_file.read()).decode('utf-8')
        return f"data:image/png;base64,{base64_data}"

file_path = 'academic.png'
image= image_to_base64_data_uri(file_path)

messages = [
    {"role": "system", "content": "You are an assistant who perfectly describes images."},
    {
        "role": "user",
        "content": [
            {"type": "image", "image": {"url": image}},
            {"type" : "text", "text": "Describe this image in detail please."}
        ]
    }
]

response = llm.create_chat_completion(messages)
print(response)

输出结果 #

{'id': 'chatcmpl-7b3fac95-4fc1-4d1c-a89e-b536331c3f57', 'object': 'chat.completion', 'created': 1749274274, 'model': 'Holo1-7B.i1-Q5_K_M.gguf', 'choices': [{'index': 0, 'message': {'role': 'assistant', 'content': "The image shows a person with short, light brown hair wearing a white t-shirt with a graphic design on the front. The design appears to be a stylized illustration or logo. The person is standing against a plain, light-colored background. The lighting is bright, highlighting the person's features and the details of the t-shirt design. The overall style is casual and modern."}, 'logprobs': None, 'finish_reason': 'stop'}], 'usage': {'prompt_tokens': 32, 'completion_tokens': 75, 'total_tokens': 107}}

排查步骤 #

• 升级llama-cpp-python版本
  旧版本对多模态模型支持不完善，先升级到最新版：

  pip install --upgrade llama-cpp-python
  

• 修正图像输入格式
  模型不支持data URI格式，改为传递纯base64字符串，同时调整调用方式：

  # 修改图像编码函数
  def image_to_base64(file_path):
      with open(file_path, "rb") as img_file:
          return base64.b64encode(img_file.read()).decode('utf-8')
  
  image = image_to_base64('academic.png')
  
  # 使用支持多模态的聊天补全格式
  response = llm.create_chat_completion(
      messages=[
          {"role": "user", "content": "<image>\nDescribe this image in detail please."}
      ],
      images=[image]
  )
  

• 调整模型加载参数
  扩大上下文窗口确保能容纳图像嵌入，同时确认多模态支持启用：

  llm = Llama(
      model_path='Holo1-7B.i1-Q5_K_M.gguf',
      n_gpu_layers=-1,
      n_ctx=4096  # 根据模型要求调整
  )
  

• 匹配模型原生prompt格式
  改用模型要求的纯文本prompt测试，避免chat模板不兼容：

  prompt = "USER: <image>\nDescribe this image in detail.\nASSISTANT:"
  response = llm(
      prompt=prompt,
      images=[image],
      max_tokens=200,
      stop=["USER:"]
  )
  print(response['choices'][0]['text'])
  

• 排除图像本身问题
  用特征明确的测试图（如猫、汽车）替换academic.png，验证模型是否能正常识别，排除原图像模糊、格式异常等问题。

内容的提问来源于stack exchange，提问作者Abhash Rai