Kafka KRaft集群单Broker启动失败：Raft IO线程异常排查

问题现象 #

• 部署3节点Kafka Broker集群（KRaft模式），任意两个节点可正常启动，但随机有一个节点启动失败，抛出错误：Unexpected error in raft IO thread (org.apache.kafka.server.fault.ProcessTerminatingFaultHandler)
• 故障节点不固定，重新部署时可能是Broker2或Broker3，但始终仅一个节点异常
• 3个Controller节点均正常运行，Broker与Controller、Controller之间通信正常

故障节点日志（kafka-broker-1） #

# docker logs kafka-broker-1
===> User
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
===> Configuring ...
Running in KRaft mode...
===> Running preflight checks ...
===> Check if /var/lib/kafka/data is writable ...
===> Running in KRaft mode, skipping Zookeeper health check...
===> Using provided cluster id mX-qLvc-T2y2OPeJ3AMRXg ...
===> Launching ...
===> Launching kafka ...
[2023-11-03 08:08:25,973] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2023-11-03 08:08:26,365] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2023-11-03 08:08:26,554] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2023-11-03 08:08:26,559] INFO [BrokerServer id=1] Transition from SHUTDOWN to STARTING (kafka.server.BrokerServer)
[2023-11-03 08:08:26,560] INFO [SharedServer id=1] Starting SharedServer (kafka.server.SharedServer)
[2023-11-03 08:08:26,656] INFO [LogLoader partition=__cluster_metadata-0, dir=/tmp/kraft-broker-logs] Loading producer state till offset 0 with message format version 2 (kafka.log.UnifiedLog$)
[2023-11-03 08:08:26,657] INFO [LogLoader partition=__cluster_metadata-0, dir=/tmp/kraft-broker-logs] Reloading from producer snapshot and rebuilding producer state from offset 0 (kafka.log.UnifiedLog$)
[2023-11-03 08:08:26,657] INFO [LogLoader partition=__cluster_metadata-0, dir=/tmp/kraft-broker-logs] Producer state recovery took 0ms for snapshot load and 0ms for segment recovery from offset 0 (kafka.log.UnifiedLog$)
[2023-11-03 08:08:26,717] INFO Initialized snapshots with IDs SortedSet() from /tmp/kraft-broker-logs/__cluster_metadata-0 (kafka.raft.KafkaMetadataLog$)
[2023-11-03 08:08:26,760] INFO [raft-expiration-reaper]: Starting (kafka.raft.TimingWheelExpirationService$ExpiredOperationReaper)
[2023-11-03 08:08:26,975] INFO [RaftManager id=1] Completed transition to Unattached(epoch=0, voters=[2, 3], electionTimeoutMs=1903) from null (org.apache.kafka.raft.QuorumState)
[2023-11-03 08:08:26,981] INFO [kafka-1-raft-outbound-request-thread]: Starting (kafka.raft.RaftSendThread)
[2023-11-03 08:08:26,981] INFO [kafka-1-raft-io-thread]: Starting (kafka.raft.KafkaRaftManager$RaftIoThread)
[2023-11-03 08:08:27,013] INFO [BrokerServer id=1] Starting broker (kafka.server.BrokerServer)
[2023-11-03 08:08:27,015] INFO [MetadataLoader id=1] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader)
[2023-11-03 08:08:27,035] INFO [broker-1-ThrottledChannelReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2023-11-03 08:08:27,035] INFO [broker-1-ThrottledChannelReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2023-11-03 08:08:27,037] INFO [broker-1-ThrottledChannelReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2023-11-03 08:08:27,039] INFO [broker-1-ThrottledChannelReaper-ControllerMutation]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2023-11-03 08:08:27,077] INFO [BrokerServer id=1] Waiting for controller quorum voters future (kafka.server.BrokerServer)
[2023-11-03 08:08:27,077] INFO [BrokerServer id=1] Finished waiting for controller quorum voters future (kafka.server.BrokerServer)
[2023-11-03 08:08:27,091] INFO [broker-1-to-controller-forwarding-channel-manager]: Starting (kafka.server.BrokerToControllerRequestThread)
[2023-11-03 08:08:27,115] INFO [MetadataLoader id=1] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader)
[2023-11-03 08:08:27,215] INFO [MetadataLoader id=1] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader)
[2023-11-03 08:08:27,282] INFO [RaftManager id=1] Registered the listener org.apache.kafka.image.loader.MetadataLoader@308714136 (org.apache.kafka.raft.KafkaRaftClient)
[2023-11-03 08:08:27,317] INFO [MetadataLoader id=1] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader)
[2023-11-03 08:08:27,417] INFO [MetadataLoader id=1] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader)
[2023-11-03 08:08:27,487] ERROR Encountered fatal fault: Unexpected error in raft IO thread (org.apache.kafka.server.fault.ProcessTerminatingFaultHandler)
java.lang.IllegalStateException: Received request or response with leader OptionalInt[1] and epoch 18 which is inconsistent with current leader OptionalInt.empty and epoch 0
    at org.apache.kafka.raft.KafkaRaftClient.maybeTransition(KafkaRaftClient.java:1513)
    at org.apache.kafka.raft.KafkaRaftClient.maybeHandleCommonResponse(KafkaRaftClient.java:1473)
    at org.apache.kafka.raft.KafkaRaftClient.handleFetchResponse(KafkaRaftClient.java:1071)
    at org.apache.kafka.raft.KafkaRaftClient.handleResponse(KafkaRaftClient.java:1550)
    at org.apache.kafka.raft.KafkaRaftClient.handleInboundMessage(KafkaRaftClient.java:1676)
    at org.apache.kafka.raft.KafkaRaftClient.poll(KafkaRaftClient.java:2251)
    at kafka.raft.KafkaRaftManager$RaftIoThread.doWork(RaftManager.scala:64)
    at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:127)

Broker1的docker-compose配置 #

---
version: '2'
services:

  broker:
    image: confluentinc/cp-kafka:7.5.1
    hostname: vskafka-broker-1
    container_name: kafka-broker-1
    ports:
      - "9092:9092"
      - "29092:29092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT'
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://vskafka-broker-1:29092'
      KAFKA_PROCESS_ROLES: 'broker'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '2@vskafka-controller-2:9093,3@vskafka-controller-3:9093'
      KAFKA_LISTENERS: 'PLAINTEXT://vskafka-broker-1:9092'
      KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_LOG_DIRS: '/tmp/kraft-broker-logs'
      CLUSTER_ID: 'mX-qLvc-T2y2OPeJ3AMRXg'

Controller1的docker-compose配置 #

---
version: '2'
services:

  controller:
    image: confluentinc/cp-kafka:7.5.1
    hostname: vskafka-controller-1
    container_name: kafka-controller-1
    ports:
      - "9093:9093"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: 'controller'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@vskafka-controller-1:9093,2@vskafka-controller-2:9093,3@vskafka-controller-3:9093'
      KAFKA_LISTENERS: 'CONTROLLER://vskafka-controller-1:9093'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_LOG_DIRS: '/tmp/kraft-controller-logs'
      CLUSTER_ID: 'mX-qLvc-T2y2OPeJ3AMRXg'

问题分析 #

核心报错java.lang.IllegalStateException: Received request or response with leader OptionalInt[1] and epoch 18 which is inconsistent with current leader OptionalInt.empty and epoch 0表明：故障Broker节点的Raft状态（epoch=0，无leader）与Controller返回的最新集群元数据（leader=1，epoch=18）不匹配。结合配置和现象，原因有两点：

1. Broker的Controller Quorum配置不完整：当前Broker仅配置了Controller2和3，缺少Controller1，导致无法同步完整的Controller集群元数据
2. Broker元数据日志目录存在旧数据残留：重启时未清理之前的元数据日志，导致节点启动时加载旧的Raft状态，无法与最新的Controller集群同步

解决方案 #

1. 修正Broker的Controller Quorum配置 #

将所有Broker的KAFKA_CONTROLLER_QUORUM_VOTERS改为包含全部3个Controller节点：

KAFKA_CONTROLLER_QUORUM_VOTERS: '1@vskafka-controller-1:9093,2@vskafka-controller-2:9093,3@vskafka-controller-3:9093'

2. 清理Broker节点的旧元数据 #

• 停止故障Broker容器：docker stop kafka-broker-1（替换为对应故障容器名）
• 删除容器：docker rm kafka-broker-1
• 若使用了持久化卷，需手动清理卷内数据：docker volume rm <对应卷名>；若未挂载卷，临时目录会自动清空，可直接跳过此步骤

3. 统一Broker节点配置 #

确保每个Broker的KAFKA_NODE_ID唯一，KAFKA_ADVERTISED_LISTENERS和KAFKA_LISTENERS与自身hostname一致，避免网络寻址错误

4. 重新部署 #

优先确保Controller集群完成leader选举，再启动所有Broker节点，保证元数据同步正常

内容的提问来源于stack exchange，提问作者Udayendu