Spring Cloud Stream Kafka自动反序列化与DLQ配置故障排查

Problem 2: Fix DLQ Configuration #

DLQ failures in Kafka Streams typically come from missing exception handler setup or incorrect topic configuration. Here's how to fix it:

Key Configuration Changes

1. Enable DLQ-aware exception handling: Use Spring's built-in handler to route failures to DLQ.
2. Specify DLQ topic and producer properties:

spring:
  cloud:
    stream:
      kafka:
        streams:
          binder:
            configuration:
              # Use Spring's DLQ-enabled exception handler
              default.production.exception.handler: org.springframework.cloud.stream.binder.kafka.streams.KafkaStreamsUncaughtExceptionHandler
              spring.cloud.stream.kafka.streams.binder.error-handler-mode: dlq
          bindings:
            processReport-in-0:
              consumer:
                dlq-name: your-dlq-topic
                dlq-producer-properties:
                  bootstrap.servers: your-kafka-bootstrap:9092
                  key.serializer: org.apache.kafka.common.serialization.StringSerializer
                  value.serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
                  schema.registry.url: http://your-schema-registry:8081

Verification Steps

1. Check DLQ Topic: Ensure the DLQ topic exists (enable Kafka's auto.create.topics.enable=true or create manually with appropriate partitions/replication).
2. Permissions: Confirm your application has WRITE access to the DLQ topic.
3. Debug Logging: Enable trace logging for DLQ operations:

   logging:
     level:
       org.springframework.cloud.stream.binder.kafka.streams: DEBUG
   

4. Test Faulty Records: Send a malformed Avro record or one that triggers a processing exception to your input topic, then check the DLQ for the failed message.
5. Avoid Exception Swallowing: Ensure your function doesn't catch and suppress exceptions—they need to propagate to the binder's handler to trigger DLQ routing.

内容的提问来源于stack exchange，提问作者Serhii