Spring Boot Kafka发送Avro数据遇Unrecognized field 'timestamp'异常求助

问题背景 #

在Spring Boot应用中调试Kafka功能，使用Avro定义了自定义Charts Schema：

{
  "type": "record",
  "namespace": "at.wolf.music.service.schema",
  "name": "Charts",
  "fields": [
    {
      "name": "page",
      "type": "int"
    },
    {
      "name": "tracks",
      "type": {
        "type": "array",
        "items": {
          "name": "Track",
          "type": "record",
          "fields": [
            {
              "name": "trackName",
              "type": "string"
            },
            {
              "name": "artistName",
              "type": "string"
            },
            {
              "name": "playcount",
              "type": "int"
            },
            {
              "name": "listeners",
              "type": "int"
            }
          ]
        }
      }
    }
  ]
}

执行kafkaChartsTemplate.send("topic1", charts);发送数据时，抛出如下异常：

Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: org.apache.kafka.common.errors.SerializationException: Error retrieving Avro schema: {"type":"record","name":"Charts","namespace":"at.wolf.music.service.schema","fields":[{"name":"page","type":"int"},{"name":"tracks","type":{"type":"array","items":{"type":"record","name":"Track","fields":[{"name":"trackName","type":{"type":"string","avro.java.string":"String"}},{"name":"artistName","type":{"type":"string","avro.java.string":"String"}},{"name":"playcount","type":"int"},{"name":"listeners","type":"int"}]}}}]}] with root cause

io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Unrecognized field "timestamp" (class io.confluent.kafka.schemaregistry.client.rest.entities.ErrorMessage), not marked as ignorable (2 known properties: "error_code", "message"])
 at [Source: (sun.net.www.protocol.http.HttpURLConnection$HttpInputStream); line: 1, column: 111] (through reference chain: io.confluent.kafka.schemaregistry.client.rest.entities.ErrorMessage["timestamp"]); error code: 50005
    at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:230) ~[kafka-schema-registry-client-5.3.0.jar:na]
    ...（省略中间堆栈）
    at java.base/java.lang.Thread.run(Thread.java:1589) ~[na:na]

使用的Kafka生产者配置如下：

@Configuration
public class KafkaProducerConfig {

    @Value("${kafka.schema.registry.url}")
    private String schemaRegistryUrl;

    @Bean
    public ProducerFactory<String, Charts> producerFactory() {
        return new DefaultKafkaProducerFactory<>(producerConfigs());
    }

    @Bean
    public KafkaTemplate<String, Charts> stringTemplate(ProducerFactory<String, Charts> pf) {
        return new KafkaTemplate<>(pf);
    }


    @Bean
    public Map<String, Object> producerConfigs() {
        Map<String, Object> props = new HashMap<>();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer.class);
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer.class);
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class);
        props.put(KafkaAvroSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
        props.put(KafkaAvroSerializerConfig.AUTO_REGISTER_SCHEMAS, false);
        return props;
    }
}

关于"timestamp"字段的来源及异常原因 #

1. 字段来源：这个timestamp字段是Schema Registry服务端返回的错误响应中包含的字段，并非你的Avro Schema或业务数据里的字段。
2. 异常本质：你使用的kafka-schema-registry-client版本是5.3.0，这个版本的ErrorMessage实体类只定义了error_code和message两个属性，没有包含timestamp字段；而你的Schema Registry服务端版本相对较新，返回的错误响应里新增了timestamp字段，导致客户端的Jackson反序列化时无法识别该字段，抛出异常。
3. 触发场景：因为你配置了AUTO_REGISTER_SCHEMAS=false，生产者发送数据时会先去Schema Registry查找对应subject的已注册Schema，但该Schema并未提前注册，服务端返回错误响应，从而触发了这个序列化问题。另外，你的生产者配置中还错误添加了消费者专属的反序列化配置，这属于冗余配置，和当前异常无关，但建议移除。

解决方案 #

• 升级Schema Registry客户端版本：将kafka-schema-registry-client和kafka-avro-serializer的版本升级到与Schema Registry服务端匹配的版本（建议至少升级到6.0及以上版本，这些版本的ErrorMessage类已经兼容timestamp字段）。
• 移除冗余配置：删除生产者配置中的ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG和ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG配置项，生产者只需要配置序列化类。
• 手动注册Schema：由于关闭了自动注册，需要提前将Charts Schema手动注册到Schema Registry的对应subject下（默认subject名称为topic1-value）。

内容的提问来源于stack exchange，提问作者mackesmilian