K8s中StatefulSet部署的第二个Apache NiFi节点绑定集群失败

在Kubernetes环境中通过StatefulSet部署2个Apache NiFi节点，单个节点运行正常，但第二个节点nifi-1启动失败，Pod日志如下：

2022-12-21 18:48:16,852 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
java.io.IOException: Failed to bind to nifi-0.nifi.namespacenifi-01.svc.cluster.local/10.233.79.58:8080
 at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
 at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
 at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
 at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
 at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
 at org.eclipse.jetty.server.Server.doStart(Server.java:401)
 at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
 at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:812)
 at org.apache.nifi.NiFi.<init>(NiFi.java:172)
 at org.apache.nifi.NiFi.<init>(NiFi.java:83)
 at org.apache.nifi.NiFi.main(NiFi.java:332)
Caused by: java.net.BindException: Cannot assign requested address
 at sun.nio.ch.Net.bind0(Native Method)
 at sun.nio.ch.Net.bind(Net.java:461)
 at sun.nio.ch.Net.bind(Net.java:453)
 at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222)
 at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85)
 at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
 ... 10 common frames omitted
2022-12-21 18:48:16,853 INFO [Thread-1] org.apache.nifi.NiFi Application Server shutdown started

附NiFi的StatefulSet配置文件：

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nifi
  labels:
    name: nifi
    app: nifi
  annotations:
    app.kubernetes.io/name: nifi
    app.kubernetes.io/part-of: nifi
spec:
  serviceName: nifi
  replicas: 2
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      app: nifi
  template:
    metadata:
      labels:
        app: nifi
    spec:
      automountServiceAccountToken: false
      enableServiceLinks: false
      restartPolicy: Always
      securityContext:
        runAsGroup: 1000
        runAsUser: 1000
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: nifi
        image: XXX
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          name: nifi
        - containerPort: 8082
          name: cluster
        env:
        - name: "NIFI_SENSITIVE_PROPS_KEY"
          value: "nificluster"
        - name: NIFI_WEB_HTTP_HOST
          valueFrom:
             fieldRef:
               fieldPath: status.podIP
        - name: NIFI_WEB_HTTP_PORT
          value: "8080"
        - name: NIFI_ANALYTICS_PREDICT_ENABLED
          value: "true"
        - name: NIFI_ELECTION_MAX_CANDIDATES
          value: "2"
        - name: NIFI_ELECTION_MAX_WAIT
          value: "1 min"
        - name: NIFI_CLUSTER_IS_NODE
          value: "true"
        - name: NIFI_JVM_HEAP_INIT
          value: "3g"
        - name: NIFI_JVM_HEAP_MAX
          value: "4g"
        - name: NIFI_CLUSTER_NODE_CONNECTION_TIMEOUT
          value: "2 min"
        - name: NIFI_CLUSTER_PROTOCOL_CONNECTION_HANDSHAKE_TIMEOUT
          value: "2 min"        
        - name: NIFI_CLUSTER_NODE_PROTOCOL_MAX_THREADS
          value: "15"
        - name: NIFI_CLUSTER_NODE_PROTOCOL_PORT
          value: "8082"
        - name: NIFI_CLUSTER_NODE_READ_TIMEOUT
          value: "15"
        - name: NIFI_ZK_CONNECT_STRING
          value: "zookeeper:2181"
        - name: NIFI_CLUSTER_PROTOCOL_IS_SECURE
          value: "false"
        - name: NIFI_CLUSTER_NODE_ADDRESS
          valueFrom:
             fieldRef:
               fieldPath: status.podIP
        # - name: HOSTNAME
        #   valueFrom:
        #     fieldRef:
        #       fieldPath: status.podIP
        livenessProbe:
          exec:
            command:
              - pgrep
              - java
          initialDelaySeconds: 60
          periodSeconds: 30
          timeoutSeconds: 10
          failureThreshold: 3
          successThreshold: 1
        readinessProbe:
          exec:
            command:
              - pgrep
              - java
          initialDelaySeconds: 180
          periodSeconds: 30
          timeoutSeconds: 10
          failureThreshold: 3
          successThreshold: 1
        resources:
          requests:
            cpu: 400m
            memory: 1Gi
          limits:
            cpu: 500m
            memory: 2Gi
      volumes:
        - name: pv--01
          persistentVolumeClaim:
            claimName: pv-claim

排查分析 #

从日志可见，nifi-1尝试绑定到nifi-0的Pod IP（10.233.79.58）的8080端口，触发BindException，核心问题在于节点的配置存储与网络标识冲突：

1. 共享存储导致配置覆盖：所有Pod挂载同一个PVC，NiFi的节点配置文件（如nifi.properties）会被不同节点互相覆盖，导致nifi-1读取到nifi-0的绑定地址配置。
2. 网络标识配置不匹配：StatefulSet的Pod有固定主机名（nifi-0、nifi-1），但当前配置仅用Pod IP作为集群地址，未利用StatefulSet的DNS域名特性，节点无法正确识别自身身份。
3. 探针失效：仅检查Java进程是否存在，无法判断NiFi服务是否就绪，节点未完全启动时就可能被纳入集群，加剧配置混乱。

解决方案 #

1. 为每个节点分配独立存储 #

替换共享PVC为volumeClaimTemplates，让每个Pod自动创建独立存储：

spec:
  volumeClaimTemplates:
  - metadata:
      name: nifi-storage
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi

同时删除原volumes配置，在容器中添加挂载：

containers:
- name: nifi
  # ...其他配置
  volumeMounts:
  - name: nifi-storage
    mountPath: /opt/nifi/nifi-current/data

2. 修正NiFi网络标识配置 #

添加环境变量，利用StatefulSet的主机名特性：

env:
# ...原有配置
- name: NIFI_CLUSTER_NODE_ADDRESS
  valueFrom:
    fieldRef:
      fieldPath: metadata.name  # 使用Pod主机名nifi-0/nifi-1
- name: NIFI_CLUSTER_NODE_PROTOCOL_ADDRESS
  value: "$(NIFI_CLUSTER_NODE_ADDRESS):8082"
- name: NIFI_WEB_PROXY_HOST
  value: "$(NIFI_CLUSTER_NODE_ADDRESS).nifi.svc.cluster.local"

3. 优化探针配置 #

替换为检查NiFi API的就绪状态：

livenessProbe:
  httpGet:
    path: /nifi-api/flow/status
    port: nifi
  initialDelaySeconds: 120
  periodSeconds: 30
  timeoutSeconds: 10
readinessProbe:
  httpGet:
    path: /nifi-api/flow/status
    port: nifi
  initialDelaySeconds: 60
  periodSeconds: 15
  timeoutSeconds: 5

修改后重新部署StatefulSet，每个节点将拥有独立存储和正确的网络标识，解决端口绑定冲突问题。

内容的提问来源于stack exchange，提问作者lbened