请求排查Kafka消费者组因组协调器无响应导致会话超时、撤销分配并重新加入组的问题
请求排查Kafka消费者组因组协调器无响应导致会话超时、撤销分配并重新加入组的问题
我把问题标题简化了一下,完整的错误提示是这样的:
Kafka Consumer group session timed out (in join-state steady) after X ms without a successful response from the group coordinator: revoking assignment and rejoining group
有没有大佬能帮忙分析下这是什么原因导致的?
上下文
我写了个简单的Python脚本,用来验证Kafka集群的数据迁移是否成功,逻辑很简单:
- 数据已经从旧Kafka集群完整复制到新集群了
- 脚本会启动两个消费者,分别连接新旧两个集群,按顺序读取对应主题的消息
- 对比两个消费者读到的数据是否完全一致,以此验证迁移是否成功
之前我还写过一个类似的迁移脚本,那个脚本只有一个消费者和一个生产者,每读取一条消息就刷生产者的缓冲区,然后提交消费者的偏移量,全程没出过任何问题。这次的脚本逻辑差不多,只是多了一个消费者用来对比数据,却遇到了这个超时问题,实在有点懵。
详细日志片段
下面是更具体的日志内容,能看到很多超时和断开的记录:
%5|1739052194.708|REQTMOUT|consumer2.topicname#consumer-2| [thrd:GroupCoordinator]: GroupCoordinator/1: Timed out OffsetCommitRequest in flight (after 158ms, timeout #0) %5|1739052194.708|REQTMOUT|consumer2.topicname#consumer-2| [thrd:GroupCoordinator]: GroupCoordinator/1: Timed out OffsetCommitRequest in flight (after 158ms, timeout #1) %5|1739052194.708|REQTMOUT|consumer2.topicname#consumer-2| [thrd:GroupCoordinator]: GroupCoordinator/1: Timed out OffsetCommitRequest in flight (after 158ms, timeout #2) %5|1739052194.708|REQTMOUT|consumer2.topicname#consumer-2| [thrd:GroupCoordinator]: GroupCoordinator/1: Timed out OffsetCommitRequest in flight (after 158ms, timeout #3) %5|1739052194.708|REQTMOUT|consumer2.topicname#consumer-2| [thrd:GroupCoordinator]: GroupCoordinator/1: Timed out OffsetCommitRequest in flight (after 158ms, timeout #4) %4|1739052194.834|REQTMOUT|consumer2.topicname#consumer-2| [thrd:GroupCoordinator]: GroupCoordinator/1: Timed out 2216 in-flight, 0 retry-queued, 15394 out-queue, 1 partially-sent requests %3|1739052194.835|FAIL|consumer2.topicname#consumer-2| [thrd:GroupCoordinator]: GroupCoordinator: 192.168.0.2:9092: 17610 request(s) timed out: disconnect (average rtt 159.627ms) (after 100166ms in state UP) %4|1739052205.056|SESSTMOUT|consumer2.topicname#consumer-2| [thrd:main]: Consumer group session timed out (in join-state steady) after 45000 ms without a successful response from the group coordinator (broker 1, last error was Local: Timed out in queue): revoking assignment and rejoining group %4|1739052206.391|REQTMOUT|consumer2.topicname#consumer-2| [thrd:GroupCoordinator]: GroupCoordinator/1: Timed out 0 in-flight, 0 retry-queued, 1 out-queue, 0 partially-sent requests %3|1739052206.392|FAIL|consumer2.topicname#consumer-2| [thrd:GroupCoordinator]: GroupCoordinator: 192.168.0.2:9092: 1 request(s) timed out: disconnect (average rtt 156.853ms) (after 9708ms in state UP) %4|1739052212.861|REQTMOUT|kafka_topic_data_verify_consumer1_rightmove.property_data#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/3: Timed out 0 in-flight, 0 retry-queued, 1 out-queue, 0 partially-sent requests %3|1739052212.862|FAIL|kafka_topic_data_verify_consumer1_rightmove.property_data#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: 192.168.0.3:9092: 1 request(s) timed out: disconnect (average rtt 146.758ms) (after 118194ms in state UP)
脚本是用Python写的,但我感觉问题应该和编程语言关系不大,主要是日志里全是OffsetCommitRequest超时、组协调器连接断开的记录,实在搞不懂为啥会这样。有没有遇到过类似问题的朋友给点排查思路?
备注:内容来源于stack exchange,提问作者user2138149
相关产品推荐
相关产品推荐

