K8s中Flink基于ZooKeeper的HA部署：Web UI leader选举故障排查

问题背景 #

在Kubernetes环境中部署了2个JobManager实例和5个ZooKeeper节点，JobManager Pod均正常启动，但Flink Web UI持续提示：

service temporarily unavailable due to an ongoing leader election. please refresh.

已执行以下操作但问题未解决：

• 重启ZooKeeper集群
• 删除ZK中/flink/i-project-dev2节点
• 重启JobManager实例
  所有操作后日志未显示明显错误。

当前配置 #

Flink 核心配置 #

jobmanager.rpc.address: some-us-east-1-dev2-project-jobmanager
high-availability: zookeeper
high-availability.zookeeper.quorum: zk-0.zk-headless.flink.svc.cluster.local:2181,zk-1.zk-headless.flink.svc.cluster.local:2181,zk-2.zk-headless.flink.svc.cluster.local:2181,zk-3.zk-headless.flink.svc.cluster.local:2181,zk-4.zk-headless.flink.svc.cluster.local:2181
high-availability.zookeeper.path.root: /flink
high-availability.cluster-id: /i-project-dev2
high-availability.storageDir: s3a://a-some-events-useast1-admin/ha/project/dev2/
high-availability.jobmanager.port: 6150
high-availability.zookeeper.client.acl: open
zookeeper.sasl.disable: true
jobmanager.rpc.port: 6123

ZooKeeper 配置（zoo.cfg） #

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/bitnami/zookeeper/data
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
autopurge.purgeInterval=24

## Metrics Providers
#
# https://prometheus.io Metrics Exporter
metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
metricsProvider.httpPort=9141
metricsProvider.exportJvmInfo=true
preAllocSize=65536
snapCount=100000
maxCnxns=0
reconfigEnabled=false
quorumListenOnAllIPs=false
4lw.commands.whitelist=srvr, mntr, ruok, stat
maxSessionTimeout=60000
admin.serverPort=8080
admin.enableServer=true
server.1=zk-0.zk-headless.flink.svc.cluster.local:2888:3888;2181
server.2=zk-1.zk-headless.flink.svc.cluster.local:2888:3888;2181
server.3=zk-2.zk-headless.flink.svc.cluster.local:2888:3888;2181
server.4=zk-3.zk-headless.flink.svc.cluster.local:2888:3888;2181
server.5=zk-4.zk-headless.flink.svc.cluster.local:2888:3888;2181

排查与解决步骤 #

• 验证网络连通性
  进入任意JobManager Pod，测试与所有ZK节点的2181端口连通性：

  nc -zv zk-0.zk-headless.flink.svc.cluster.local 2181
  nc -zv zk-1.zk-headless.flink.svc.cluster.local 2181
  # 依次测试剩余3个ZK节点
  

  同时测试JobManager之间的RPC端口互通：

  nc -zv some-us-east-1-dev2-project-jobmanager 6123
  nc -zv <另一个JobManager的Pod IP> 6150
  

• 检查HA存储权限
  确认JobManager对S3存储目录有读写权限：

  # 列出存储目录内容
  hadoop fs -ls s3a://a-some-events-useast1-admin/ha/project/dev2/
  # 创建测试文件验证写入权限
  hadoop fs -touchz s3a://a-some-events-useast1-admin/ha/project/dev2/test-leader-check
  

  若操作失败，检查Pod绑定的IAM角色或密钥是否配置正确。

• 修正Flink配置异常
  修改high-availability.cluster-id配置，移除开头的斜杠（Flink不推荐该配置以斜杠开头，会导致ZK路径层级异常）：

  high-availability.cluster-id: i-project-dev2
  

  同时确认jobmanager.rpc.address对应的K8s Service selector与JobManager Pod标签完全匹配。

• 检查ZK集群状态
  进入任意ZK Pod，使用客户端命令查看集群状态：

  zkCli.sh
  # 查看ZK集群状态
  stat
  # 查看Flink HA路径下的leader节点
  ls /flink/i-project-dev2/leader
  

  执行4lw命令确认ZK节点角色与quorum状态：

  echo srvr | nc localhost 2181
  

  确保至少3个ZK节点处于正常状态（5节点集群需过半节点在线）。

• 开启细粒度日志排查
  调整JobManager的log4j配置，开启leader选举与ZK相关的DEBUG日志：

  <Logger name="org.apache.flink.runtime.leaderelection" level="DEBUG" />
  <Logger name="org.apache.flink.runtime.highavailability.zookeeper" level="DEBUG" />
  

  重启JobManager后，重点关注日志中关于ZK锁注册、存储目录读写的细节，定位选举阻塞点。

• 强制重置HA状态
  若上述步骤无效，执行完全重置：

  1. 停止所有JobManager实例
  2. 清理ZK中Flink的根路径：

     zkCli.sh rmr /flink
     

  3. 清理S3存储目录下的所有HA数据：

     hadoop fs -rm -r s3a://a-some-events-useast1-admin/ha/project/dev2/*
     

  4. 重新启动ZooKeeper与JobManager集群

内容的提问来源于stack exchange，提问作者hitesh