使用@xenova/transformers加载Moondream2模型时遇protobuf解析失败错误

问题重现 #

使用@xenova/transformers运行Moondream2模型时触发如下错误：

Error: Failed to load model because protobuf parsing failed.
    at new OnnxruntimeSessionHandler (/root/moondream/node_modules/onnxruntime-node/dist/backend.js:27:92)
    at Immediate.<anonymous> (/root/moondream/node_modules/onnxruntime-node/dist/backend.js:64:29)
    at process.processImmediate (node:internal/timers:483:21)

运行环境：Node.js v22.4.0、Ubuntu 24.04 LTS，执行代码见下方示例。

核心原因及解决方案 #

1. 模型类与模型ID不匹配 #

代码中使用Moondream1ForConditionalGeneration加载Xenova/moondream2模型，类版本与模型版本不兼容，直接导致加载逻辑错误。

修正操作：
将Moondream1ForConditionalGeneration替换为Moondream2ForConditionalGeneration：

import {
  AutoProcessor,
  AutoTokenizer,
  Moondream2ForConditionalGeneration, // 替换为Moondream2对应的类
  RawImage,
} from "@xenova/transformers";

// ... 其余代码不变
const model = await Moondream2ForConditionalGeneration.from_pretrained(
  model_id,
  {
    dtype: {
      embed_tokens: "fp16",
      vision_encoder: "fp16",
      decoder_model_merged: "q4",
    },
    device: "cpu",
  }
);

2. 模型缓存文件损坏 #

自动下载的模型文件可能因网络中断、磁盘异常等原因损坏，引发protobuf解析失败。

操作步骤：

• 删除transformers默认缓存目录：

  rm -rf ~/.cache/huggingface/transformers
  

• 重新运行代码，让库重新下载完整的模型文件。

3. onnxruntime-node与Node.js 22兼容性问题 #

Node.js 22属于较新版本，部分onnxruntime-node版本存在兼容性冲突，导致protobuf解析流程出错。

操作步骤：

• 卸载当前版本的onnxruntime-node：

  npm uninstall onnxruntime-node
  

• 安装兼容的稳定版本（如1.16.3）：

  npm install onnxruntime-node@1.16.3
  

4. 系统缺少Protobuf依赖库 #

Ubuntu 24.04可能缺少protobuf相关的系统级依赖，导致onnxruntime无法正常解析protobuf格式文件。

操作步骤：

• 安装libprotobuf-dev：

  sudo apt update && sudo apt install libprotobuf-dev
  

完整修正后的代码示例 #

import {
  AutoProcessor,
  AutoTokenizer,
  Moondream2ForConditionalGeneration,
  RawImage,
} from "@xenova/transformers";

// Load processor, tokenizer and model
const model_id = "Xenova/moondream2";
try {
  const processor = await AutoProcessor.from_pretrained(model_id);

  const tokenizer = await AutoTokenizer.from_pretrained(model_id);

  const model = await Moondream2ForConditionalGeneration.from_pretrained(
    model_id,
    {
      dtype: {
        embed_tokens: "fp16", // or 'fp32'
        vision_encoder: "fp16", // or 'q8'
        decoder_model_merged: "q4", // or 'q4f16' or 'q8'
      },
      device: "cpu",
    }
  );

  // Prepare text inputs
  const prompt = "Describe this image.";
  const text = `<image>\n\nQuestion: ${prompt}\n\nAnswer:`;
  const text_inputs = tokenizer(text);

  // Prepare vision inputs
  const url =
    "https://huggingface.co/vikhyatk/moondream1/resolve/main/assets/demo-1.jpg";
 
  const image = await RawImage.fromURL(url);
  const vision_inputs = await processor(image);

  // Generate response
  const output = await model.generate({
    ...text_inputs,
    ...vision_inputs,
    do_sample: false,
    max_new_tokens: 64,
  });
  const decoded = tokenizer.batch_decode(output, { skip_special_tokens: false });
  console.log("answer new", decoded);
} catch (error) {
  console.log("error", error);
}

内容的提问来源于stack exchange，提问作者Abhishek MG