Kafka KRaft环境下NotLeaderException问题排查与预防咨询

背景说明 #

此前我看到过类似问题，但我的场景不同，我们使用的是Kafka KRaft而非依赖Zookeeper的Kafka。

环境规格 #

Kafka version: 3.3.1
Number of brokers: 8
Minimum replication factor of topics: 3

问题描述 #

当前该问题已多次出现，Kafka日志如下：

[2023-01-09 09:53:03,929] WARN [Controller 3] maybeFenceReplicas: failed with unknown server exception NotLeaderException at epoch 2641 in 1913 us.  Renouncing leadership and reverting to the last committed offset 9986340. (org.apache.kafka.controller.QuorumController)
org.apache.kafka.raft.errors.NotLeaderException: Append failed because the replication is not the current leader
        at org.apache.kafka.raft.KafkaRaftClient.lambda$append$27(KafkaRaftClient.java:2262)
        at java.base/java.util.Optional.orElseThrow(Optional.java:408)
        at org.apache.kafka.raft.KafkaRaftClient.append(KafkaRaftClient.java:2261)
        at org.apache.kafka.raft.KafkaRaftClient.scheduleAtomicAppend(KafkaRaftClient.java:2257)
        at org.apache.kafka.controller.QuorumController$ControllerWriteEvent$1.apply(QuorumController.java:813)
        at org.apache.kafka.controller.QuorumController$ControllerWriteEvent$1.apply(QuorumController.java:792)
        at org.apache.kafka.controller.QuorumController.appendRecords(QuorumController.java:903)
        at org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:791)
        at org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:121)
        at org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:200)
        at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:173)
        at java.base/java.lang.Thread.run(Thread.java:829)
[2023-01-09 09:53:03,931] INFO [Controller 3] writeNoOpRecord: failed with NotControllerException in 415741179 us (org.apache.kafka.controller.QuorumController)
[2023-01-09 09:53:03,931] INFO [Controller 3] writeNoOpRecord: failed with NotControllerException in 206629449 us (org.apache.kafka.controller.QuorumController)
[2023-01-09 09:53:03,931] INFO [Controller 3] maybeFenceReplicas: failed with NotControllerException in 206629220 us (org.apache.kafka.controller.QuorumController)
[2023-01-09 09:53:03,931] INFO [Controller 3] maybeFenceReplicas: failed with NotControllerException in 206626538 us (org.apache.kafka.controller.QuorumController)
[2023-01-09 09:53:03,931] INFO [Controller 3] maybeFenceReplicas: failed with NotControllerException in 205746648 us (org.apache.kafka.controller.QuorumController)
[2023-01-09 09:53:03,931] INFO [Controller 3] maybeFenceReplicas: failed with NotControllerException in 7549 us (org.apache.kafka.controller.QuorumController)
[2023-01-09 09:53:03,931] INFO [Controller 3] maybeFenceReplicas: failed with NotControllerException in 6986 us (org.apache.kafka.controller.QuorumController)
[2023-01-09 09:53:03,931] INFO [Controller 3] maybeFenceReplicas: failed with NotControllerException in 6399 us (org.apache.kafka.controller.QuorumController)
[2023-01-09 09:53:03,931] INFO [Controller 3] maybeFenceReplicas: failed with NotControllerException in 5912 us (org.apache.kafka.controller.QuorumController)
[2023-01-09 09:53:03,932] ERROR [Controller 3] Unexpected exception while executing deferred write event maybeFenceReplicas. Rescheduling for a minute from now. (org.apache.kafka.controller.QuorumController)
org.apache.kafka.common.errors.UnknownServerException: org.apache.kafka.raft.errors.NotLeaderException: Append failed because the replication is not the current leader
Caused by: org.apache.kafka.raft.errors.NotLeaderException: Append failed because the replication is not the current leader
        at org.apache.kafka.raft.KafkaRaftClient.lambda$append$27(KafkaRaftClient.java:2262)
        at java.base/java.util.Optional.orElseThrow(Optional.java:408)
        at org.apache.kafka.raft.KafkaRaftClient.append(KafkaRaftClient.java:2261)
        at org.apache.kafka.raft.KafkaRaftClient.scheduleAtomicAppend(KafkaRaftClient.java:2257)
        at org.apache.kafka.controller.QuorumController$ControllerWriteEvent$1.apply(QuorumController.java:813)
        at org.apache.kafka.controller.QuorumController$ControllerWriteEvent$1.apply(QuorumController.java:792)
        at org.apache.kafka.controller.QuorumController.appendRecords(QuorumController.java:903)
        at org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:791)
        at org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:121)
        at org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:200)
        at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:173)
        at java.base/java.lang.Thread.run(Thread.java:829)

还有：

ERROR [Controller 3] processBrokerHeartbeat: unable to start processing because of NotControllerException. (org.apache.kafka.controller.QuorumController)

这是生产节点，我们用Prometheus和Grafana持续监控。时间戳显示该broker在2023-01-09 09:53出现故障，监控显示其他7个broker应正常工作且不应发生数据丢失，但监控结果与预期不符。

该问题在11:31再次出现。

疑问 #

根据监控截图和主题消息，我认为未发生数据丢失，该判断是否正确？如何防止该问题再次发生？

解答 #

数据丢失判断结论

你的判断是正确的，未发生数据丢失。
日志显示Broker 3在执行控制器操作时发现自己已不是当前Raft Leader，主动放弃控制器角色并回退到已提交的偏移量，属于控制器角色切换的正常异常流程，而非数据副本损坏或丢失。结合你设置的最小副本因子3，其余7个Broker正常运行时，主题的副本集合仍能保持可用性，Kafka Raft协议会保证已提交的数据不会丢失。只要监控中未出现ISR集合收缩至小于副本因子、副本同步中断的情况，数据就是安全的。

防止问题再次发生的措施

• 升级Kafka版本：Kafka 3.3.1是KRaft早期稳定版本，存在Raft控制器选举、角色切换相关的已知Bug，建议升级到3.4.0及以上版本，官方后续版本修复了多个KRaft稳定性问题。
• 优化Raft控制器配置：
  • 调大controller.quorum.timeout.ms：默认3000ms，若集群网络存在抖动，可调整为5000-10000ms，避免短暂网络延迟触发不必要的控制器重选举。
  • 合理设置raft.heartbeat.interval.ms：建议设为500ms，确保心跳间隔远小于超时时间，保证控制器节点状态同步及时。
• 强化监控体系：
  • 新增监控指标：跟踪kafka_controller_is_active（节点是否为活跃控制器）、kafka_raft_leader_epoch（Raft Leader的Epoch变化频率），频繁的Epoch切换说明集群稳定性存在隐患。
  • 监控Broker网络状态：重点关注Broker 3与其他控制器节点的网络延迟、丢包率，排查是否存在网络硬件或链路故障。
• 优化控制器节点分布：
  • 确保KRaft控制器节点（controller.quorum.voters配置）分布在不同物理机或可用区，避免单点故障或局部网络问题影响控制器集群。
• 优化Broker资源：
  • 检查Broker 3的磁盘IO负载，磁盘读写延迟过高会导致控制器事件队列堆积，触发角色切换异常。建议使用低延迟存储（如SSD），定期清理日志文件。

内容的提问来源于stack exchange，提问作者Mostafa Ghadimi