Kafka KRaft集群控制器节点10分钟后断开连接问题咨询

• 开发环境部署KRaft模式Kafka集群，包含3个兼具controller和broker角色的节点，集群初期运行正常，但10分钟后节点1日志显示节点2和节点3已断开连接
• 各节点日志表现：
  • 节点1：记录节点2、3断开连接
  • 节点2：无节点断开相关日志
  • 节点3：记录节点2断开连接

节点1日志（部署阶段WARN信息可忽略） #

[2023-04-14 15:18:41,618] INFO [SocketServer listenerType=BROKER, nodeId=1] Enabling request processing. (kafka.network.SocketServer)
[2023-04-14 15:18:41,621] INFO [StandardAuthorizer 1] Completed initial ACL load process. (org.apache.kafka.metadata.authorizer.StandardAuthorizerData)
[2023-04-14 15:18:41,626] INFO [Controller 1] The request from broker 1 to unfence has been granted because it has caught up with the offset of it's register broker record 1. (org.apache.kafka.controller.BrokerHeartbeatManager)
[2023-04-14 15:18:41,646] INFO [Controller 1] Unfenced broker: 1 (org.apache.kafka.controller.ClusterControlManager)
[2023-04-14 15:18:41,647] INFO [RaftManager nodeId=1] Node 3 disconnected. (org.apache.kafka.clients.NetworkClient)
[2023-04-14 15:18:41,648] WARN [RaftManager nodeId=1] Connection to node 3 (/172.31.38.35:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2023-04-14 15:18:41,700] INFO [BrokerLifecycleManager id=1] The broker has been unfenced. Transitioning from RECOVERY to RUNNING. (kafka.server.BrokerLifecycleManager)
[2023-04-14 15:18:41,701] INFO [BrokerServer id=1] Transition from STARTING to STARTED (kafka.server.BrokerServer)
[2023-04-14 15:18:41,703] INFO Kafka version: 3.3.1 (org.apache.kafka.common.utils.AppInfoParser)
[2023-04-14 15:18:41,703] INFO Kafka commitId: e23c59d00e687ff5 (org.apache.kafka.common.utils.AppInfoParser)
[2023-04-14 15:18:41,703] INFO Kafka startTimeMs: 1681478321701 (org.apache.kafka.common.utils.AppInfoParser)
[2023-04-14 15:18:41,704] INFO [KafkaRaftServer nodeId=1] Kafka Server started (kafka.server.KafkaRaftServer)
[2023-04-14 15:18:41,778] INFO [Controller 1] The request from broker 2 to unfence has been granted because it has caught up with the offset of it's register broker record 3. (org.apache.kafka.controller.BrokerHeartbeatManager)
[2023-04-14 15:18:41,778] INFO [Controller 1] Unfenced broker: 2 (org.apache.kafka.controller.ClusterControlManager)
[2023-04-14 15:18:42,103] INFO [RaftManager nodeId=1] Node 3 disconnected. (org.apache.kafka.clients.NetworkClient)
[2023-04-14 15:18:42,103] WARN [RaftManager nodeId=1] Connection to node 3 (/172.31.38.35:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2023-04-14 15:18:42,658] INFO [RaftManager nodeId=1] Node 3 disconnected. (org.apache.kafka.clients.NetworkClient)
[2023-04-14 15:19:18,348] INFO [Controller 1] Added new fenced broker: 3 (org.apache.kafka.controller.ClusterControlManager)
[2023-04-14 15:19:18,349] INFO [Controller 1] Registered new broker: RegisterBrokerRecord(brokerId=3, incarnationId=X_1FgQvvQkSE8ybWbaKWWg, brokerEpoch=80, endPoints=[BrokerEndpoint(name='SSL', host='172.31.38.35', port=9094, securityProtocol=1)], features=[BrokerFeature(name='metadata.version', minSupportedVersion=1, maxSupportedVersion=7)], rack=null, fenced=true, inControlledShutdown=false) (org.apache.kafka.controller.ClusterControlManager)
[2023-04-14 15:19:18,818] INFO [Controller 1] The request from broker 3 to unfence has been granted because it has caught up with the offset of it's register broker record 80. (org.apache.kafka.controller.BrokerHeartbeatManager)
[2023-04-14 15:19:18,819] INFO [Controller 1] Unfenced broker: 3 (org.apache.kafka.controller.ClusterControlManager)
[2023-04-14 15:28:41,407] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2023-04-14 15:29:17,391] INFO [RaftManager nodeId=1] Node 3 disconnected. (org.apache.kafka.clients.NetworkClient)

节点1的server.properties配置 #

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@172.31.46.13:9093,2@172.31.35.247:9093,3@172.31.38.35:9093

listeners=CONTROLLER://:9093,SSL://:9094
advertised.listeners=SSL://172.31.46.13:9094
controller.listener.names=CONTROLLER
security.inter.broker.protocol=SSL

ssl.enabled.protocols=TLSv1.3,TLSv1.2,TLSv1.1,TLSv1
ssl.client.auth=required
ssl.keystore.location=/opt/kafka/certs/kafka-01.keystore.jks
ssl.keystore.password=
ssl.key.password=
ssl.truststore.location=/opt/kafka/certs/server.truststore.jks
ssl.truststore.password=

authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
super.users=User:kafka-01;User:kafka-02;User:kafka-03
ssl.principal.mapping.rules=RULE:^CN=(.*?),OU=Test,O=Test.*$/$1/

这种INFO级别的节点断开日志不属于集群正常稳定运行的表现，核心原因大概率是controller节点间的通信配置或网络问题，以下是具体排查方向和修复方案：

1. CONTROLLER监听器SSL配置缺失
   当前配置中security.inter.broker.protocol=SSL，但CONTROLLER监听器未指定安全协议，导致controller节点间（9093端口）尝试用SSL通信时缺少必要的密钥/信任库配置，最终握手失败触发断开日志。
   修复：添加监听器安全协议映射，确保CONTROLLER监听器使用SSL：

   listener.security.protocol.map=CONTROLLER:SSL,SSL:SSL
   

   同时给所有节点的CONTROLLER监听器配置对应的SSL参数（密钥库、信任库路径及密码），与SSL监听器保持一致。

2. 网络超时参数不合理
   开发环境可能存在网络波动，默认的Raft会话超时、连接超时过短会导致心跳中断。可调整以下参数延长超时时间：

   controller.quorum.session.timeout.ms=30000
   controller.quorum.connection.timeout.ms=10000
   

   同时用nc -zv 172.31.38.35 9093持续测试节点间9093端口的连通性，确认无网络阻塞。

3. SSL证书与Principal验证问题
   虽然配置了双向SSL认证，但需确认所有节点的证书都已加入集群信任库，且ssl.principal.mapping.rules能正确解析出super.users中配置的Principal（如kafka-01）。证书验证失败会直接断开连接，且部分节点可能不记录详细错误。
   临时开启DEBUG日志排查：

   log4j.logger.org.apache.kafka.clients.NetworkClient=DEBUG
   

   查看连接断开时的SSL握手错误详情。

4. KRaft版本兼容性
   使用的Kafka 3.3.1版本中KRaft模式仍存在部分已知的稳定性问题，升级到3.4及以上版本可修复一些Raft通信相关的bug，建议尝试版本升级验证。

内容的提问来源于stack exchange，提问作者eazy-b