无需Hazelcast Operator，OpenShift容器中Hazelcast选主集群能否运行？

问题背景 #

本地环境下（同一IP、不同端口运行2个节点）集成Hazelcast Leader Election的SpringBoot微服务可正常工作，但部署到OpenShift后，不同IP、相同端口的节点无法互相发现，出现集群脑裂问题。受限于环境无法安装Hazelcast Operator、配置RBAC规则或修改OpenShift环境设置，需通过应用级编码或YAML配置解决。

核心代码及Maven依赖如下：

@Component
public class Runner implements CommandLineRunner, MembershipListener {
    private static final Logger logger = Logger.getLogger(Runner.class.getName());
...
    private String leaderAddress;
    private HazelcastInstance hazelcastInstance;
...    
    @Autowired
    private CSPRabbitMQConfig rabbitMQConfiguration;

    public Runner(RabbitTemplate rabbitTemplate) {
        this.rabbitTemplate = rabbitTemplate;
    }
    @Override
    public void run(String... args) throws Exception {      
        hazelcastInstance = Hazelcast.newHazelcastInstance(/* hazelcastConfig */);
        hazelcastInstance.getCluster().addMembershipListener(this);
        
        logger.info("Members: "+ hazelcastInstance.getCluster().getMembers().size());
        leaderAddress = hazelcastInstance.getCluster().getMembers().iterator().next().getSocketAddress().toString();
        logger.info("Current leader address: " + leaderAddress);
...
    // Leader should be the only process to start the subscriptions
    if (amITheLeader()) {
        // TODO: need to verify this works in a multi-host environment.
        // This node is the leader, do leader-specific tasks here
...
        }
...
    }
    ...
    public boolean amITheLeader() {
        logger.info("> amITheLeader() - cluster size: " + hazelcastInstance.getCluster().getMembers().size());
        return hazelcastInstance.getCluster().getLocalMember().getSocketAddress().toString().equals(leaderAddress);
    }

    @Override
    public void memberAdded(MembershipEvent membershipEvent) {
        if (membershipEvent.getMember().getSocketAddress().toString().compareTo(leaderAddress) < 0) {
            // A new member has joined that has an older address, it becomes the new leader
            leaderAddress = membershipEvent.getMember().getSocketAddress().toString();
            if (amITheLeader()) {
                // This node is the new leader, do leader-specific tasks here
            }
        }
    }

    @Override
    public void memberRemoved(MembershipEvent membershipEvent) {
        if (membershipEvent.getMember().getSocketAddress().toString().equals(leaderAddress)) {
            // The leader has left the cluster, a new leader will be elected
            leaderAddress = hazelcastInstance.getCluster().getMembers().iterator().next().getSocketAddress().toString();
            if (amITheLeader()) {
                // This node is the new leader, do leader-specific tasks here
            }                  
        }
    }

Maven依赖：

<dependency>
            <groupId>com.hazelcast</groupId>
            <artifactId>hazelcast-spring</artifactId>
        </dependency>

解决方案 #

1. 替换节点发现方式为TCP/IP（禁用多播） #

OpenShift默认禁用多播，必须改用TCP/IP模式，通过OpenShift Service的DNS名称实现节点互发现。修改run方法中的Hazelcast配置：

@Override
public void run(String... args) throws Exception {      
    Config config = new Config();
    NetworkConfig networkConfig = config.getNetworkConfig();
    
    // 禁用多播（OpenShift环境不支持）
    networkConfig.getJoin().getMulticastConfig().setEnabled(false);
    
    // 启用TCP/IP发现，指定OpenShift Service名称（替换为你的服务名）
    TcpIpConfig tcpIpConfig = networkConfig.getJoin().getTcpIpConfig();
    tcpIpConfig.setEnabled(true);
    tcpIpConfig.addMember("my-springboot-service"); // Service名称
    tcpIpConfig.setConnectionTimeout(5000); // 连接超时时间
    
    // 设置Pod公网地址，避免容器内部IP访问问题
    String podIp = System.getenv("POD_IP");
    if (podIp != null) {
        networkConfig.setPublicAddress(podIp + ":5701");
    }
    
    hazelcastInstance = Hazelcast.newHazelcastInstance(config);
    hazelcastInstance.getCluster().addMembershipListener(this);
    
    // 使用Hazelcast内置Leader获取方法，替代手动维护的leaderAddress
    Member leader = hazelcastInstance.getCluster().getLeader();
    logger.info("Current leader address: " + leader.getSocketAddress());
    ...
}

2. 简化Leader选举逻辑（使用Hazelcast内置API） #

你当前手动维护leaderAddress的方式容易出错，Hazelcast已内置可靠的Leader选举机制，直接使用即可：

public boolean amITheLeader() {
    Member currentLeader = hazelcastInstance.getCluster().getLeader();
    Member localMember = hazelcastInstance.getCluster().getLocalMember();
    logger.info("> amITheLeader() - cluster size: " + hazelcastInstance.getCluster().getMembers().size());
    return localMember.equals(currentLeader);
}

@Override
public void memberAdded(MembershipEvent membershipEvent) {
    Member newLeader = hazelcastInstance.getCluster().getLeader();
    logger.info("New member added, current leader: " + newLeader.getSocketAddress());
    if (amITheLeader()) {
        // 执行Leader专属任务
    }
}

@Override
public void memberRemoved(MembershipEvent membershipEvent) {
    Member newLeader = hazelcastInstance.getCluster().getLeader();
    logger.info("Member removed, new leader: " + newLeader.getSocketAddress());
    if (amITheLeader()) {
        // 执行Leader专属任务
    }                  
}

3. OpenShift Deployment YAML配置 #

确保暴露Hazelcast端口，并传递Pod IP环境变量：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-springboot-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-springboot-app
  template:
    metadata:
      labels:
        app: my-springboot-app
    spec:
      containers:
      - name: my-springboot-container
        image: your-image:tag
        ports:
        - containerPort: 5701  # Hazelcast默认端口
        - containerPort: 8080  # SpringBoot应用端口
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
---
apiVersion: v1
kind: Service
metadata:
  name: my-springboot-service
spec:
  selector:
    app: my-springboot-app
  ports:
  - name: hazelcast
    port: 5701
    targetPort: 5701
  - name: http
    port: 8080
    targetPort: 8080

说明 #

• 你的手动Leader选举实现确实较为原始，依赖SocketAddress字符串比较容易出现逻辑漏洞，使用Hazelcast内置的getLeader()方法更可靠。
• TCP/IP发现通过OpenShift Service的DNS解析，可自动获取所有后端Pod的IP，无需手动维护成员列表。
• 设置publicAddress是为了让Hazelcast节点对外暴露正确的Pod IP，避免容器内部网络访问限制导致的连接失败。

内容的提问来源于stack exchange，提问作者Paul M