含Timestamp字段的Proto消息编码失败问题求助

问题场景 #

在TypeScript中使用kafkajs和@kafkajs/confluent-schema-registry，通过Proto编码消息发送至Kafka时，出现以下错误：

/Users/test-user/Documents/Workspace/typescript-producer/node_modules/@kafkajs/confluent-schema-registry/node_modules/protobufjs/src/namespace.js:411
        throw Error("no such Type or Enum '" + path + "' in " + this);
              ^
Error: no such Type or Enum 'google.protobuf.Timestamp' in Type .kafka.data_tests.v0.User
    at Type.lookupTypeOrEnum (/Users/test-user/Documents/Workspace/typescript-producer/node_modules/@kafkajs/confluent-schema-registry/node_modules/protobufjs/src/namespace.js:411:15)
    at Field.resolve (/Users/test-user/Documents/Workspace/typescript-producer/node_modules/@kafkajs/confluent-schema-registry/node_modules/protobufjs/src/field.js:268:94)
    at Type.setup (/Users/test-user/Documents/Workspace/typescript-producer/node_modules/@kafkajs/confluent-schema-registry/node_modules/protobufjs/src/type.js:435:41)
    at Type.verify_setup [as verify] (/Users/test-user/Documents/Workspace/typescript-producer/node_modules/@kafkajs/confluent-schema-registry/node_modules/protobufjs/src/type.js:529:17)
    at ProtoSchema.isValid (/Users/test-user/Documents/Workspace/typescript-producer/node_modules/@kafkajs/confluent-schema-registry/src/ProtoSchema.ts:64:48)
    at SchemaRegistry.collectInvalidPaths (/Users/test-user/Documents/Workspace/typescript-producer/node_modules/@kafkajs/confluent-schema-registry/src/SchemaRegistry.ts:210:12)
    at SchemaRegistry.encode (/Users/test-user/Documents/Workspace/typescript-producer/node_modules/@kafkajs/confluent-schema-registry/src/SchemaRegistry.ts:203:26)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async main (/Users/test-user/Documents/Workspace/typescript-producer/producer-with-schema-code.ts:67:28)

使用的Proto schema：

syntax = "proto3";
package kafka.data_tests.v0;

import "google/protobuf/timestamp.proto";

message User {
  string name = 1;
  string surname = 2;
  bool employee = 3;
  google.protobuf.Timestamp created_at = 4;
}

已安装的依赖：

"devDependencies": {
    "@kafkajs/confluent-schema-registry": "^3.2.1",
    "@types/google-protobuf": "^3.15.6",
    "@types/node": "^14.11.2",
    "@types/uuid": "^8.3.4",
    "google-protobuf": "^3.20.1",
    "gts": "^3.1.0",
    "kafkajs": "^2.1.0",
    "protobufjs": "^7.0.0",
    "ts-protoc-gen": "^0.15.0",
    "typescript": "^4.7.4",
    "uuid": "^8.3.2"
  }

解决方案 #

• 确保导入的Timestamp proto文件可被加载
  从protobuf官方仓库复制google/protobuf目录到你的项目proto文件所在目录（比如src/proto/google/protobuf），这样proto文件中的import "google/protobuf/timestamp.proto"能被正确解析。

• 配置SchemaRegistry的proto根目录
  创建SchemaRegistry实例时，指定proto文件的根目录，让它能找到依赖的proto文件：

  import { SchemaRegistry } from '@kafkajs/confluent-schema-registry';
  import { readFileSync } from 'fs';
  
  const registry = new SchemaRegistry({
    host: 'http://your-registry-host:8081',
    protofile: {
      root: './src/proto', // 你的proto根目录，包含google/protobuf子目录
    },
  });
  
  // 加载并注册schema
  const userSchemaContent = readFileSync('./src/proto/user.proto', 'utf8');
  const schemaId = await registry.register({ type: 'PROTOBUF', schema: userSchemaContent });
  

• 修复依赖版本冲突
  当前使用的protobufjs@7.0.0与@kafkajs/confluent-schema-registry@3.2.1内部依赖的protobufjs版本存在兼容性问题，降级protobufjs到6.x版本：
  修改package.json中的依赖：

  "protobufjs": "^6.11.3"
  

  执行npm install或yarn install重新安装依赖。

• 确保TS类型文件包含Timestamp定义
  如果使用ts-protoc-gen生成TypeScript类型，生成时指定proto根目录，确保Timestamp的类型被正确生成：

  protoc --plugin=protoc-gen-ts=./node_modules/.bin/protoc-gen-ts \
    --ts_out=./src/generated \
    --proto_path=./src/proto \
    ./src/proto/user.proto
  

内容的提问来源于stack exchange，提问作者alerojlop