NiFi集群搭建求助：服务运行但GUI无法访问

环境信息 #

Java版本 #

openjdk version "1.8.0_302"
OpenJDK Runtime Environment (build 1.8.0_302-b08)
OpenJDK 64-Bit Server VM (build 25.302-b08, mixed mode)

NiFi版本 #

nifi-1.17.0

nifi.properties配置 #

nifi.state.management.embedded.zookeeper.start=true
nifi.remote.input.host=Svxxx.xyz.com
nifi.remote.input.secure=true
nifi.remote.input.socket.port=10443
nifi.remote.input.http.enabled=true
nifi.web.https.host=Svxxx.xyz.com
nifi.web.https.port=9443
nifi.web.proxy.host=localhost:9443,Svxxx.xyz.com:9443
nifi.sensitive.props.key=propkeywith12chars
nifi.cluster.is.node=true
nifi.cluster.node.address=Svxxx.xyz.com
nifi.cluster.node.protocol.port=11443
nifi.cluster.load.balance.host=Svxxx.xyz.com
nifi.cluster.load.balance.port=6342
nifi.zookeeper.connect.string=Svxxx.xyz.com:2181,Svxxx.xyz.com:2181,Svxxx.xyz.com:2181

zookeeper.properties配置 #

server.1=Svxxx.xyz.com:2888:3888;2181
server.2=Svxxx.xyz.com:2888:3888;2181
server.3=Svxxx.xyz.com:2888:3888;2181

state-management.xml修改内容 #

<cluster-provider>
<id>zk-provider</id>
<class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
<property name="Connect String">Svxxx.xyz.com:2181,Svxxx.xyz.com:2181,Svxxx.xyz.com:2181</property>
<property name="Root Node">/nifi</property>
<property name="Session Timeout">10 seconds</property>
<property name="Access Control">Open</property>
</cluster-provider>

其他配置信息 #

• 防火墙状态：已禁用
• SSL证书：已使用工具生成并部署，替换了truststore.jks和keystore.jks

日志信息（nifi-app.log） #

2022-10-17 22:00:05,669 WARN [main] o.a.nifi.controller.StandardFlowService 当前没有集群协调器。这种情况通常在运行嵌入式ZooKeeper的NiFi重启时发生。将注册此节点成为活动集群协调器，并尝试重新连接集群
2022-10-17 22:00:05,670 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] 尝试为'Cluster Coordinator'角色注册领导者选举，但该角色已注册
2022-10-17 22:00:11,403 WARN [Heartbeat Monitor Thread-1] o.a.n.c.l.e.CuratorLeaderElectionManager 无法确定'Cluster Coordinator'角色的领导者；返回null
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /nifi/leaders/Cluster Coordinator
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:2480)
        at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:242)
        at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:231)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:93)
        at org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:228)
        at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:219)
        at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.locks.LockInternals.getSortedChildren(LockInternals.java:154)
        at org.apache.curator.framework.recipes.locks.LockInternals.getParticipantNodes(LockInternals.java:134)
        at org.apache.curator.framework.recipes.locks.InterProcessMutex.getParticipantNodes(InterProcessMutex.java:170)
        at org.apache.curator.framework.recipes.leader.LeaderSelector.getLeader(LeaderSelector.java:337)
        at org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.getLeader(CuratorLeaderElectionManager.java:281)
        at org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener.verifyLeader(CuratorLeaderElectionManager.java:571)
        at org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener.isLeader(CuratorLeaderElectionManager.java:525)
        at org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$LeaderRole.isLeader(CuratorLeaderElectionManager.java:466)
        at org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.isLeader(CuratorLeaderElectionManager.java:262)
        at org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.isActiveClusterCoordinator(NodeClusterCoordinator.java:824)
        at org.apache.nifi.cluster.coordination.heartbeat.AbstractHeartbeatMonitor.monitorHeartbeats(AbstractHeartbeatMonitor.java:132)
        at org.apache.nifi.cluster.coordination.heartbeat.AbstractHeartbeatMonitor$1.run(AbstractHeartbeatMonitor.java:84)
        at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2022-10-17 22:00:12,371 INFO [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository 成功从归档中删除0个文件（0字节）
2022-10-17 22:00:12,371 INFO [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository 容器default的归档清理完成；现在允许写入此容器。已用字节=10.53 GB，可用字节=26.46 GB，容量=36.99 GB
2022-10-17 22:00:14,203 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@22e813fc 已检查点，包含1条记录和0个交换文件，耗时5毫秒（停顿时间=2毫秒，清除编辑日志时间=2毫秒），最大事务ID 3
2022-10-17 22:00:18,823 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager 无法确定'Cluster Coordinator'角色的领导者；返回null
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /nifi/leaders/Cluster Coordinator

问题分析与解决建议 #

核心问题定位 #

日志反复出现KeeperException$ConnectionLossException，说明NiFi节点无法连接ZooKeeper集群，导致集群协调器选举失败，进而影响GUI正常服务。

具体排查步骤 #

1. 检查ZooKeeper集群状态

   • 执行zkServer.sh status查看每个ZooKeeper节点状态（leader/follower），确认节点正常启动
   • 在NiFi节点上执行telnet Svxxx.xyz.com 2181、telnet Svxxx.xyz.com 2888、telnet Svxxx.xyz.com 3888，验证端口连通性
   • 查看ZooKeeper日志（zookeeper.out），排查连接异常、节点初始化失败等错误

2. 修正ZooKeeper配置错误
   当前zookeeper.properties中三台服务器使用同一域名，无法组成有效集群。需：

   • 为三台ZooKeeper节点分配唯一主机名/IP地址
   • 修改配置文件，确保每个server.X指向不同节点，示例：

     server.1=zk-node1.xyz.com:2888:3888;2181
     server.2=zk-node2.xyz.com:2888:3888;2181
     server.3=zk-node3.xyz.com:2888:3888;2181
     

   • 在每个ZooKeeper节点的dataDir目录下创建myid文件，内容为对应节点编号（1、2、3）

3. 修正NiFi的ZooKeeper连接配置

   • 更新nifi.properties中的nifi.zookeeper.connect.string，使用正确的三台ZooKeeper节点地址
   • 更新state-management.xml中的Connect String配置，与上述保持一致

4. 验证SSL证书配置

   • 确认nifi.properties中SSL相关配置（nifi.security.keystore、nifi.security.truststore等）指向正确路径，且NiFi进程用户有读取权限
   • 执行keytool -list -keystore keystore.jks验证证书有效性，确保证书主机名与nifi.web.https.host匹配

5. 重启服务并验证

   • 先停止所有NiFi节点，再停止所有ZooKeeper节点
   • 按顺序启动ZooKeeper节点，确认集群选举出leader
   • 启动NiFi节点，查看日志是否仍有连接异常，待集群协调器选举成功后，访问https://Svxxx.xyz.com:9443验证GUI可用性

内容的提问来源于stack exchange，提问作者priyanka