Kafka数天后出现Auto offset commit失败及UnknownMemberIdError问题求助

部署了3节点Kafka集群，每个微服务针对特定Topic使用独立的Group ID。与Kafka通信数天后，总会出现异常，报错信息如下：

Unable connect to node with id 1:
Failed fetch messages from 1: NodeNotReadyError: Attempt to send a request to node which is not ready (node id 1).
Failed fetch messages from 2: [Error 7] RequestTimedOutError
Failed fetch messages from 1: [Error 7] RequestTimedOutError
Failed fetch messages from 2: [Error 7] RequestTimedOutError
Error sending JoinGroupRequest_v2 to node 1 [[Error 7] RequestTimedOutError] -- marking coordinator dead
Marking the coordinator dead (node 1)for group _message_alarm_ticket_app1.
Failed fetch messages from 3: [Error 7] RequestTimedOutError
Heartbeat failed: local member_id was not recognized; resetting and re-joining group
Heartbeat session expired - marking coordinator dead
Marking the coordinator dead (node 3)for group nce_alarms.
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
Auto offset commit failed: [Error 25] UnknownMemberIdError: nce_alarms

Topic描述 #

root@dev-s-kafka1:/opt/kafka/bin# ./kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic nce_alarms
Topic: nce_alarms       TopicId: zDniSSlUTgS4bWyXKPg5Zw PartitionCount: 8       ReplicationFactor: 3    Configs: segment.bytes=1073741824
        Topic: nce_alarms       Partition: 0    Leader: 3       Replicas: 3,1,2 Isr: 3,2,1
        Topic: nce_alarms       Partition: 1    Leader: 1       Replicas: 1,2,3 Isr: 3,2,1
        Topic: nce_alarms       Partition: 2    Leader: 2       Replicas: 2,3,1 Isr: 2,3,1
        Topic: nce_alarms       Partition: 3    Leader: 3       Replicas: 3,2,1 Isr: 3,2,1
        Topic: nce_alarms       Partition: 4    Leader: 1       Replicas: 1,3,2 Isr: 3,2,1
        Topic: nce_alarms       Partition: 5    Leader: 2       Replicas: 2,1,3 Isr: 2,3,1
        Topic: nce_alarms       Partition: 6    Leader: 3       Replicas: 3,1,2 Isr: 3,2,1
        Topic: nce_alarms       Partition: 7    Leader: 1       Replicas: 1,2,3 Isr: 2,3,1

环境信息 #

• aiokafka version 0.8.0
• kafka-python version 2.0.2
• Kafka Broker version 3.0.0
• Python version 3.9.16

刚接手该项目，若需更多信息可向同事咨询，恳请帮忙解决此问题。

内容的提问来源于stack exchange，提问作者Plaoo