DeadLetterPublishingRecoverer在Kafka批量消费者中的工作机制及重试死信配置疑问

Question #

I'm looking to understand how DeadLetterPublishingRecoverer works with Kafka batch consumers, since most documentation examples focus on single-record consumers. My goal is to implement retry logic for batch consumers: when retries hit a threshold (e.g., 3 times), failed records should be sent to a Dead Letter Topic (DLT) via DeadLetterPublishingRecoverer, and my business scenario doesn't require message ordering.

I have this key question:

Will DeadLetterPublishingRecoverer only send the failed records to the DLT, while the remaining successful messages in the batch continue to be processed by the consumer? Or will it send the entire batch (including successful messages) to the DLT along with the failed ones?

Here's my current configuration code:

public ConcurrentKafkaListenerContainerFactory concurrentKafkaListenerContainerFactory(KafkaTemplate<String, String> kafkaTemplate) {
    ConcurrentKafkaListenerContainerFactory<String, String> concurrentKafkaListenerContainerFactory = new ConcurrentKafkaListenerContainerFactory<>();
    concurrentKafkaListenerContainerFactory.setBatchListener(true);
    DeadLetterPublishingRecoverer deadLetterPublishingRecoverer = new DeadLetterPublishingRecoverer(kafkaTemplate);
    RecoveringBatchErrorHandler recoveringBatchErrorHandler = new RecoveringBatchErrorHandler(deadLetterPublishingRecoverer,new FixedBackOff(1000L, 3L));
    concurrentKafkaListenerContainerFactory.setBatchErrorHandler(recoveringBatchErrorHandler);
    return concurrentKafkaListenerContainerFactory;
}

Answer #

Great question—this is a common point of confusion when moving from single-record to batch Kafka consumers. Let’s break this down clearly:

Core Behavior of Your Configuration #

Only the failed records will be sent to the DLT; successful messages in the batch will be processed normally.

Here’s why this happens with your setup:

• The RecoveringBatchErrorHandler is built to handle batch processing at the individual record level. When a batch is processed and some records fail, it iterates through the batch, identifies the failed entries, and applies your retry logic (FixedBackOff(1000L, 3L)) only to those failed records.
• After 3 retries for a failed record, the DeadLetterPublishingRecoverer activates and sends that specific record to the configured DLT.
• Successful records in the same batch are acknowledged as processed correctly and won’t be touched by the error handler or dead-letter logic.

Additional Notes for Your Scenario #

Since your business doesn’t require message ordering, this setup is ideal. The error handler’s per-record retry/dead-letter behavior won’t introduce ordering issues (which would be a consideration if you needed strict sequence preservation).

If you did need ordering, you’d have to adjust to handle entire batches on failure, but that’s unnecessary here. Your current code aligns perfectly with your requirements: isolate failed records after retries, send them to DLT, and let successful messages proceed as intended.

内容的提问来源于stack exchange，提问作者Raman Preet Singh