Docker中嵌入式Hazelcast Jet集群无法组建问题求助

Let's dig into why your containers aren't forming a cluster and walk through the fixes step by step.

First, Diagnose the Core Issues #

Looking at your logs and code, here are the key problems:

1. Docker Network Misconfiguration: Your containers are on different subnets (172.19.0.3 vs 172.18.0.4), which means they can't reach each other at all. That's a showstopper for any cluster discovery method.
2. Conflicting Discovery Logic: Your code uses two separate if blocks for Multicast and TCP/IP. If both flags are enabled, the second block (TCP/IP) will overwrite the first (Multicast), leading to unexpected behavior.
3. Missing Public Address: Hazelcast nodes need to advertise a reachable address to other nodes. In Docker, the container's internal IP can change, and without setting a public address, nodes might not communicate correctly.
4. Port Auto-Increment Confusion: Since each container runs a single Jet instance, port auto-increment isn't necessary and can lead to mismatched port expectations between nodes.

Step-by-Step Fixes #

1. Fix Docker Network Setup #

First, ensure all your Jet containers are on the same custom Docker network. The default bridge network doesn't support Multicast and has limited DNS resolution. Create a custom network in your docker-compose.yml:

version: '3.8'
services:
  jet-node:
    build: .
    environment:
      # Adjust these based on your preferred discovery method
      - MULTICAST_ENABLED=false
      - TCPIP_ENABLED=true
      - TCPIP_MEMBERS=jet-node-1,jet-node-2,jet-node-3
      - PORT=5701
    networks:
      - jet-network
    deploy:
      replicas: 3 # Scale to 3 nodes

networks:
  jet-network:
    driver: bridge
    options:
      com.docker.network.bridge.enable_multicast: "true" # Enable if using Multicast

This network lets containers communicate via their service names (e.g., jet-node-1) and supports Multicast if you need it.

2. Clean Up Your Discovery Configuration Logic #

Update your code to use mutually exclusive discovery methods so one doesn't overwrite the other. Also, add the public address configuration:

JetConfig cfg = new JetConfig();
Configuration jsonObject = getJetConfigs();

// Configure base port (no auto-increment needed for single-instance containers)
int port = jsonObject.getInteger("port");
cfg.getHazelcastConfig().getNetworkConfig().setPort(port);
cfg.getHazelcastConfig().getNetworkConfig().setPortAutoIncrement(false);

// Advertise the container's hostname (resolvable in the custom Docker network)
String publicAddress = System.getenv("HOSTNAME") + ":" + port;
cfg.getHazelcastConfig().setPublicAddress(publicAddress);
LOGGER.info("Advertising public address: {}", publicAddress);

// Mutually exclusive discovery setup
boolean useMulticast = jsonObject.getBoolean("multicastEnabled");
boolean useTcpIp = jsonObject.getBoolean("tcpIpEnabled");

if (useMulticast && !useTcpIp) {
    LOGGER.info("Using Multicast discovery");
    var joinConfig = cfg.getHazelcastConfig().getNetworkConfig().getJoin();
    joinConfig.getTcpIpConfig().setEnabled(false);
    var multicastConfig = joinConfig.getMulticastConfig();
    multicastConfig.setEnabled(true);
    multicastConfig.setMulticastGroup(jsonObject.getString("multicastGroup"));
    multicastConfig.setMulticastPort(jsonObject.getInteger("multicastPort"));
} else if (useTcpIp && !useMulticast) {
    LOGGER.info("Using TCP/IP discovery");
    var joinConfig = cfg.getHazelcastConfig().getNetworkConfig().getJoin();
    joinConfig.getMulticastConfig().setEnabled(false);
    var tcpIpConfig = joinConfig.getTcpIpConfig();
    tcpIpConfig.setEnabled(true);
    // Enable DNS lookup to resolve Docker service names
    tcpIpConfig.setEnableDnsLookup(true);
    
    String tcpMembers = jsonObject.getString("tcpIpMemberList");
    List<String> memberList = tcpMembers != null ? Arrays.asList(tcpMembers.split(",")) : Collections.emptyList();
    LOGGER.info("TCP/IP member list: {}", memberList);
    tcpIpConfig.setMembers(memberList);
} else {
    LOGGER.warn("Only one discovery method should be enabled! Multicast: {}, TCP/IP: {}", useMulticast, useTcpIp);
    // Fallback to disable both (safe default)
    var joinConfig = cfg.getHazelcastConfig().getNetworkConfig().getJoin();
    joinConfig.getMulticastConfig().setEnabled(false);
    joinConfig.getTcpIpConfig().setEnabled(false);
}

3. Verify Network Connectivity #

After updating your setup, test if containers can reach each other:

• Exec into one container: docker exec -it <container-id> sh
• Ping another container's service name: ping jet-node-2
• If using TCP/IP, try telnet to the target port: telnet jet-node-2 5701

If these fail, double-check your Docker network configuration—ensure all containers are attached to the same network.

4. Troubleshoot Logs #

• If you still see nodes being blacklisted (like [172.18.0.4]:5703 is added to the blacklist), check that:
  • The public address is correctly set.
  • The member list in TCP/IP mode uses the right service names/ports.
  • No firewalls (container or host) are blocking port 5701 (or your configured port).

Final Checks #

• For Multicast: Ensure your custom Docker network has enable_multicast: "true" and your Multicast group/port matches across all nodes.
• For TCP/IP: Use Docker service names in the member list (they're resolvable in the custom network) and enable DNS lookup in Hazelcast.

With these changes, your Jet containers should discover each other and form a cluster successfully.

内容的提问来源于stack exchange，提问作者mahendra yadav