Apache Kafka 3.5无ZooKeeper集群Broker数分钟后崩溃求助

问题描述 #

使用Apache Kafka 3.5搭建3节点无ZooKeeper（KRaft）集群，所有Broker启动数分钟后均自动停止运行，节点间存在连接异常。

节点配置信息 #

三个节点的server.properties仅node.id不同，以下为节点1的配置：

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@120.201.245.119:19092,2@120.201.245.16:19093,3@120.201.245.121:19094
listeners=PLAINTEXT://:9092,CONTROLLER://:19092
inter.broker.listener.name=PLAINTEXT
advertised.listeners=PLAINTEXT://:9092
controller.listener.names=CONTROLLER
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kraft-combined-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
initial.broker.registration.timeout.ms=240000

关键错误日志 #

[2023-06-23 10:45:57,443] WARN [RaftManager id=1] Connection to node 3 (/120.201.245.121:19094) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2023-06-23 10:45:57,900] WARN [RaftManager id=1] Connection to node 3 (/120.201.245.121:19094) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2023-06-23 10:45:57,914] WARN [RaftManager id=1] Connection to node 2 (/120.201.245.16:19093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2023-06-23 10:45:58,229] WARN [RaftManager id=1] Graceful shutdown timed out after 5000ms (org.apache.kafka.raft.KafkaRaftClient)
[2023-06-23 10:45:58,230] ERROR [kafka-1-raft-io-thread]: Graceful shutdown of RaftClient failed (kafka.raft.KafkaRaftManager$RaftIoThread)
java.util.concurrent.TimeoutException: Timeout expired before graceful shutdown completed
        at org.apache.kafka.raft.KafkaRaftClient$GracefulShutdown.failWithTimeout(KafkaRaftClient.java:2423)
        at org.apache.kafka.raft.KafkaRaftClient.maybeCompleteShutdown(KafkaRaftClient.java:2169)
        at org.apache.kafka.raft.KafkaRaftClient.poll(KafkaRaftClient.java:2234)
        at kafka.raft.KafkaRaftManager$RaftIoThread.doWork(RaftManager.scala:64)
        at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:127)

环境与操作步骤 #

• 系统环境：RHEL 8.4
• 安装步骤：

  tar xzf kafka_2.13-3.5.0.tgz
  cd /home/kafka_2.13-3.5.0/config/kraft
  

• 初始化操作（每个节点修改对应node.id后执行）：

  # 生成集群ID（仅在一个节点执行，其他节点复用此ID）
  KAFKA_CLUSTER_ID="$(bash /home/kafka_2.13-3.5.0/bin/kafka-storage.sh random-uuid)"
  # 格式化存储
  bash /home/kafka_2.13-3.5.0/bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c /home/kafka_2.13-3.5.0/config/kraft/server.properties
  # 启动Broker
  bash /home/kafka_2.13-3.5.0/bin/kafka-server-start.sh -daemon /home/kafka_2.13-3.5.0/config/kraft/server.properties
  

• 已确认所有节点间可正常ping通及SSH连接。

排查与修复建议 #

1. 检查CONTROLLER端口连通性 #

日志核心问题是节点间无法建立CONTROLLER端口（19092/19093/19094）连接，需验证端口是否开放：

• 用nc -zv 目标IP 目标端口测试跨节点端口可达性，例如：

  nc -zv 120.201.245.16 19093
  

• RHEL 8.4需通过firewalld开放端口：

  # 开放当前节点的Broker和Controller端口
  firewall-cmd --add-port=9092/tcp --permanent
  firewall-cmd --add-port=19092/tcp --permanent  # 节点1用19092，节点2用19093，节点3用19094
  firewall-cmd --reload
  

2. 修正监听地址配置 #

当前advertised.listeners未指定节点IP，导致其他节点无法正确识别Broker地址：

• 每个节点需配置自身IP到advertised.listeners，例如节点1：

  advertised.listeners=PLAINTEXT://120.201.245.119:9092
  

• 同时建议在listeners中指定节点IP，避免绑定到localhost：

  listeners=PLAINTEXT://120.201.245.119:9092,CONTROLLER://120.201.245.119:19092
  

3. 确认集群ID一致性 #

所有节点必须使用相同的集群ID格式化存储，检查每个节点/tmp/kraft-combined-logs/meta.properties中的cluster.id，确保与生成的kx9G7XV3TOiagO4Fsu1FYQ一致，不一致需重新格式化。

4. 调整初始化超时时间 #

增大initial.broker.registration.timeout.ms值，给节点足够时间完成注册：

initial.broker.registration.timeout.ms=360000

5. 检查日志目录权限 #

确保/tmp/kraft-combined-logs目录有读写权限，若后续切换普通用户运行，需调整权限：

chown -R kafka:kafka /tmp/kraft-combined-logs

内容的提问来源于stack exchange，提问作者Judy