Kafka Kraft集群迁移后日志异常排查求助

3节点Kafka集群（2个同机房，1个异地）从ZooKeeper迁移至Kraft后，集群可100%访问，但出现三类异常日志，以下是各异常的分析及排查建议：

异常1：Quorum leader被设为none #

现象 #

分两种场景出现：

• 节点断开连接导致：每周随机出现几次
• 候选节点触发元数据事件导致：每小时多次出现，集中在同机房的两个节点

日志示例：

[2024-05-20 09:06:09,859] INFO [QuorumController id=1] In the new epoch 13004, the leader is (none). (org.apache.kafka.controller.QuorumController)
[2024-05-20 09:06:09,998] INFO [BrokerToControllerChannelManager id=1 name=heartbeat] Client requested disconnect from node 2 (org.apache.kafka.clients.NetworkClient)
[2024-05-20 09:06:10,449] INFO [RaftManager id=1] Completed transition to Unattached(epoch=13005, voters=[1, 2, 3], electionTimeoutMs=11) from Unattached(epoch=13004, voters=[1, 2, 3], electionTimeoutMs=628) (org.apache.kafka.raft.QuorumState)
[2024-05-20 09:06:10,449] INFO [RaftManager id=1] Vote request VoteRequestData(clusterId='ba92tKAvQY2zT-PzieD7sA', topics=[TopicData(topicName='__cluster_metadata', partitions=[PartitionData(partitionIndex=0, candidateEpoch=13005, candidateId=3, lastOffsetEpoch=13001, lastOffset=5515535)])]) with epoch 13005 is rejected (org.apache.kafka.raft.KafkaRaftClient)
[2024-05-20 09:06:10,449] INFO [QuorumController id=1] In the new epoch 13005, the leader is (none). (org.apache.kafka.controller.QuorumController) 

[2024-05-20 09:06:09,358] WARN [QuorumController id=2] Renouncing the leadership due to a metadata log event. We were the leader at epoch 13001, but in the new epoch 13002, the leader is (none). Reverting to last stable offset 5515581. (org.apache.kafka.controller.QuorumController) 

分析与排查 #

• 节点断开连接导致的leader is (none)：属于Raft协议正常的选举间隙状态，只要集群能快速选出新leader且业务无感知，就无需过度担心。但每周多次出现需排查：
  • 检查节点间网络连通性，重点关注异地节点与同机房节点的延迟、丢包率
  • 查看节点GC日志，确认是否存在长时间GC停顿（超过选举超时时间）
  • 核对controller.quorum.voters配置，确保无重复或错误节点
• 候选节点触发元数据事件导致的高频出现：需警惕，说明元数据日志频繁触发leader退位，可能是同机房节点间元数据同步异常：
  • 检查同机房节点的磁盘IO负载，看是否因IO瓶颈导致元数据日志写入延迟过高
  • 查看__cluster_metadata主题的副本同步状态，确认ISR列表是否完整
  • 调整Raft选举参数（如raft.election.timeout.ms），避免短暂延迟触发频繁选举

异常2：分区被标记为失败 #

现象 #

单个异地节点（节点3）上几乎所有主题都出现该警告，日志显示副本epoch落后于leader，需等待新的LeaderAndIsr状态再恢复拉取：

[2024-05-19 04:02:03,589] WARN [ReplicaFetcher replicaId=3, leaderId=1, fetcherId=0] Partition __consumer_offsets-40 marked as failed (kafka.server.ReplicaFetcherThread)
[2024-05-19 04:02:03,589] INFO [ReplicaFetcher replicaId=3, leaderId=1, fetcherId=0] Partition enrichment_topology_2-0 has an older epoch (67) than the current leader. Will await the new LeaderAndIsr state before resuming fetching. (kafka.server.ReplicaFetcherThread)
[2024-05-19 04:02:03,589] WARN [ReplicaFetcher replicaId=3, leaderId=1, fetcherId=0] Partition enrichment_topology_2-0 marked as failed (kafka.server.ReplicaFetcherThread)
[2024-05-19 04:02:03,590] INFO [ReplicaFetcher replicaId=3, leaderId=1, fetcherId=0] Partition __consumer_offsets-36 has an older epoch (67) than the current leader. Will await the new LeaderAndIsr state before resuming fetching. (kafka.server.ReplicaFetcherThread)
[2024-05-19 04:02:03,590] WARN [ReplicaFetcher replicaId=3, leaderId=1, fetcherId=0] Partition __consumer_offsets-36 marked as failed (kafka.server.ReplicaFetcherThread)
[2024-05-19 04:02:03,590] INFO [ReplicaFetcher replicaId=3, leaderId=1, fetcherId=0] Partition __consumer_offsets-4 has an older epoch (67) than the current leader. Will await the new LeaderAndIsr state before resuming fetching. (kafka.server.ReplicaFetcherThread)
[2024-05-19 04:02:03,590] WARN [ReplicaFetcher replicaId=3, leaderId=1, fetcherId=0] Partition __consumer_offsets-4 marked as failed (kafka.server.ReplicaFetcherThread) 

分析与排查 #

该现象是异地节点与leader节点（同机房）的epoch同步延迟导致的临时状态，但频繁出现会影响副本可用性，排查方向：

• 优先检查异地节点与同机房节点的网络带宽、延迟，确认是否存在网络拥塞
• 查看异地节点的磁盘性能指标，判断是否因IO瓶颈导致副本拉取速度跟不上leader
• 调整副本拉取参数（如replica.fetch.wait.max.ms、replica.fetch.min.bytes），优化同步效率
• 检查leader节点的CPU、内存负载，看是否因资源不足无法及时处理副本拉取请求

异常3：ZK迁移状态WARN日志 #

现象 #

日志显示加载ZK迁移状态为NONE，级别为WARN：

[2024-05-20 09:06:10,477] WARN [QuorumController id=1] Performing controller activation. Loaded ZK migration state of NONE. (org.apache.kafka.controller.QuorumController)

分析 #

该日志属于正常现象，无需担心。Kafka完成ZK到Kraft的迁移后，会保留迁移状态检查逻辑，NONE表示迁移已完成，无遗留状态需要处理。WARN级别是因为迁移逻辑需要兼容未完成迁移的场景，并非实际错误。

集群配置参考 #

Kafka堆配置 #

KAFKA_HEAP_OPTS=-Xmx6g -Xms6g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80 -XX:+ExplicitGCInvokesConcurrent

节点属性文件（已排除无关项） #

process.roles=broker,controller
quorum.type=raft
inter.broker.listener.name=PLAINTEXT
advertised.listeners=PLAINTEXT://:9092 (3rd node needs to have his IP explicitly stated there since data center resolve his host name in some strange way)
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
controller.listener.names=CONTROLLER
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
metadata.replication.factor=3
log.message.format.version=3.4
num.partitions=1
default.replication.factor=3
min.insync.replicas=2
offsets.topic.replication.factor=3
transaction.state.log.min.isr=2
transaction.state.log.replication.factor=3
ssl.cipher.suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
ssl.enabled.protocols=TLSv1.2
ssl.protocol=TLSv1.2
ssl.endpoint.identification.algorithm=HTTPS
broker.id=(1|2|3)
controller.quorum.voters=xxx
cluster.id=ba92tKAvQY2zT-PzieD7sA

内容的提问来源于stack exchange，提问作者Rafał Wójcik