Pods无法连接Gossip Router，无法形成JGroups单集群问题

问题背景 #

部署JGroups应用的StatefulSet、Gossip Router的Deployment和Service后，所有资源状态显示正常，但每个应用Pod单独形成集群，无法通过Gossip Router聚合为单集群。核心原因是应用Pod配置的jgroups.gossip_routers为localhost[12001]，导致无法连接到独立部署的Gossip Router。

相关配置与代码 #

1. StatefulSet配置 #

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: jgroups-leader-poc
  labels:
    app: jgroups-leader-poc
spec:
  serviceName: jgroups-leader-poc
  replicas: 3
  selector:
    matchLabels:
      app: jgroups-leader-poc
  template:
    metadata:
      labels:
        app: jgroups-leader-poc
    spec:
      containers:
        - name: jgroups-leader-poc-container
          image: localhost:5001/jgroups-leader-ui:1.0
          imagePullPolicy: Always
          env:
            - name: jgroups.gossip_routers
              value: "localhost[12001]"
            - name: jgroups.tcp.ip
              value: "site_local,match-interface:eth0"
            - name: jgroups.tcp.ntfnport
              value: "7800"
            - name: JGROUPS_EXTERNAL_ADDR
              value: "match-interface:eth0"
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
            - containerPort: 7800
              name: k8sping-port

2. JGroups TcpGossip.xml配置 #

<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns="urn:org:jgroups"
        xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
    <TCP external_addr="${JGROUPS_EXTERNAL_ADDR:match-interface:eth0}"
         bind_addr="${jgroups.tcp.ip}" bind_port="${jgroups.tcp.ntfnport:0}"
         sock_conn_timeout="300"
         max_bundle_size="60000"
         enable_diagnostics="false"

         thread_naming_pattern="cl"
         thread_pool.enabled="true"
         thread_pool.min_threads="1"
         thread_pool.max_threads="25"
         thread_pool.keep_alive_time="5000" />

    <TCPGOSSIP initial_hosts="${jgroups.gossip_routers:localhost[12001]}" reconnect_interval="3000"/>
    <MERGE3 min_interval="10000" max_interval="30000"/>
    <FD_SOCK/>
    <FD_ALL timeout="60000" interval="15000" timeout_check_interval="5000"/>
    <FD_HOST check_timeout="5000" interval="15000" timeout="60000"/>
    <VERIFY_SUSPECT timeout="5000"/>
    <pbcast.NAKACK2 use_mcast_xmit="false" use_mcast_xmit_req="false" xmit_interval="1000"/>
    <UNICAST3 xmit_table_max_compaction_time="3000" />
    <pbcast.STABLE desired_avg_gossip="50000" max_bytes="4M"/>
    <pbcast.GMS print_local_addr="true" join_timeout="5000" view_bundling="true"/>
    <MFC max_credits="2M"/>
    <FRAG2 frag_size="30K"/>
    <pbcast.STATE buffer_size="1048576" max_pool="10"/>
    <pbcast.FLUSH timeout="0"/>
</config>

3. Gossip Router的Deployment和Service #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gossiprouter
  labels:
    run: gossiprouter
spec:
  replicas: 1
  selector:
    matchLabels:
      run: gossiprouter
  template:
    metadata:
      labels:
        run: gossiprouter
    spec:
      containers:
        - image: belaban/gossiprouter:latest
          name: gossiprouter
          ports:
            - containerPort: 8787
            - containerPort: 9000
            - containerPort: 12001
          env:
            - name: LogLevel
              value: "TRACE"
---
apiVersion: v1
kind: Service
metadata:
  name: gossiprouter
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
    - port: 8787
      targetPort: 8787
      name: debug
      protocol: TCP
    - port: 9000
      targetPort: 9000
      name: netcat
      protocol: TCP
    - port: 12001
      targetPort: 12001
      name: gossiprouter
      protocol: TCP
  selector:
    run: gossiprouter

4. 业务代码 #

public class Chatter extends ReceiverAdapter {
    JChannel channel;
    @Value("${app.jgroups.config:jgroups-config.xml}")
    private String jGroupsConfig;

    @Value("${app.jgroups.cluster:chat-cluster}")
    private String clusterName;

    @PostConstruct
    public void init()  {
       try {
            channel = new JChannel(jGroupsConfig);
            channel.setReceiver(this);
            channel.connect(clusterName);
            checkLeaderStatus();
            channel.getState(null, 10000);
        } catch (Exception ex) {
            log.error("registering the channel in JMX failed: {}", ex);
        }
    }

    public void close() {
        channel.close();
    }

    public void viewAccepted(View newView) {
        log.info("view: " + newView);
        checkLeaderStatus();
    }

    private void checkLeaderStatus() {
        Address address = channel.getView().getMembers().get(0);
        if (address.equals(channel.getAddress())) {
            log.info("Nodes are started, I'm the master!");
        }
        else
        {
            log.info("Nodes are started, I'm a slave!");
        }
    }

}

问题分析 #

localhost[12001]表示应用Pod尝试连接自身的12001端口，但Gossip Router是独立部署的Pod，并非每个应用Pod内部都运行Gossip Router实例，因此无法建立连接，导致每个Pod只能形成独立集群。

解决方案 #

修改StatefulSet中jgroups.gossip_routers环境变量的值，使用Kubernetes Service的DNS名称访问Gossip Router：

• 如果Gossip Router的Service与StatefulSet在同一命名空间，直接使用gossiprouter[12001]
• 如果不在同一命名空间，使用完整DNS格式：gossiprouter.<命名空间>.svc.cluster.local[12001]

修改后的StatefulSet环境变量片段：

env:
  - name: jgroups.gossip_routers
    value: "gossiprouter[12001]"
  # 其余环境变量保持不变

验证步骤 #

1. 重新部署StatefulSet：

   kubectl apply -f <你的StatefulSet配置文件路径>
   

2. 查看应用Pod日志，确认是否成功连接Gossip Router并加入同一集群：

   kubectl logs -f <应用Pod名称>
   

3. 检查日志中的JGroups视图信息，确认所有Pod都在同一个集群视图内。

内容的提问来源于stack exchange，提问作者Mahendran