Spring Kafka事务偏移提交失败：消费者组元数据不匹配问题

异常日志 #

Exception:org.springframework.kafka.KafkaException: Seek to current after exception; nested exception is org.apache.kafka.clients.consumer.CommitFailedException: Transaction offset Commit failed due to consumer group metadata mismatch: The coordinator is not aware of this member.
at org.springframework.kafka.listener.SeekUtils.seekOrRecover(SeekUtils.java:206) \~\[spring-kafka-2.7.14.jar!/:2.7.14\]
at org.springframework.kafka.listener.SeekToCurrentErrorHandler.handle(SeekToCurrentErrorHandler.java:112) \~\[spring-kafka-2.7.14.jar!/:2.7.14\]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeErrorHandler(KafkaMessageListenerContainer.java:2377) \~\[spring-kafka-2.7.14.jar!/:2.7.14\]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:2254) \~\[spring-kafka-2.7.14.jar!/:2.7.14\]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.access$2300(KafkaMessageListenerContainer.java:518) \~\[spring-kafka-2.7.14.jar!/:2.7.14\]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer$3.doInTransactionWithoutResult(KafkaMessageListenerContainer.java:2092) \~\[spring-kafka-2.7.14.jar!/:2.7.14\]
at org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:36) \~\[spring-tx-5.3.20.jar!/:5.3.20\]
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) \~\[spring-tx-5.3.20.jar!/:5.3.20\]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeInTransaction(KafkaMessageListenerContainer.java:2083) \~\[spring-kafka-2.7.14.jar!/:2.7.14\]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListenerInTx(KafkaMessageListenerContainer.java:2052) \~\[spring-kafka-2.7.14.jar!/:2.7.14\]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:2028) \~\[spring-kafka-2.7.14.jar!/:2.7.14\]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:1704) \~\[spring-kafka-2.7.14.jar!/:2.7.14\]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeIfHaveRecords(KafkaMessageListenerContainer.java:1275) \~\[spring-kafka-2.7.14.jar!/:2.7.14\]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1267) \~\[spring-kafka-2.7.14.jar!/:2.7.14\]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1162) \~\[spring-kafka-2.7.14.jar!/:2.7.14\]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) \~\[?:?\]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) \~\[?:?\]
at java.lang.Thread.run(Thread.java:834) \~\[?:?\]
Caused by: org.apache.kafka.clients.consumer.CommitFailedException: Transaction offset Commit failed due to consumer group metadata mismatch: The coordinator is not aware of this member.
at org.apache.kafka.clients.producer.internals.TransactionManager$TxnOffsetCommitHandler.handleResponse(TransactionManager.java:1732) \~\[kafka-clients-2.7.2.jar!/:?\]
at org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:1290) \~\[kafka-clients-2.7.2.jar!/:?\]
at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109) \~\[kafka-clients-2.7.2.jar!/:?\]
at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:586) \~\[kafka-clients-2.7.2.jar!/:?\]
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:578) \~\[kafka-clients-2.7.2.jar!/:?\]
at org.apache.kafka.clients.producer.internals.Sender.maybeSendAndPollTransactionalRequest(Sender.java:415) \~\[kafka-clients-2.7.2.jar!/:?\]
at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:313) \~\[kafka-clients-2.7.2.jar!/:?\]
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:240) \~\[kafka-clients-2.7.2.jar!/:?\]

异常原因分析 #

该异常的核心是Kafka消费者组协调器不再识别当前消费者成员，导致事务性偏移量提交失败，具体诱因分为以下几种：

• 消费者会话超时：消费者与协调器的心跳间隔超过session.timeout.ms配置值，协调器判定该消费者已下线并将其移出组。此时消费者仍尝试提交偏移量，就会触发元数据不匹配错误。
• 频繁重平衡或消费者重启：集群分区重平衡、消费者频繁重启都会让协调器更新组内成员信息。旧的消费者实例在重平衡完成后未重新加入组就提交偏移量，会被协调器拒绝。
• 事务超时：使用事务性消费时，若事务执行时间超过transaction.timeout.ms，协调器会终止该事务。此时消费者再提交偏移量，就会因事务状态无效而失败。
• 网络波动：消费者与集群间的短暂网络中断会导致协调器收不到心跳，将消费者标记为下线。网络恢复后，消费者未重新加入组就提交偏移量，引发异常。
• 事务配置不匹配：Spring事务边界与Kafka消费偏移量提交时机不匹配，比如事务提交时消费者已被踢出组，也会触发该异常。

内容的提问来源于stack exchange，提问作者sunghyun kent lee