Docker中两个Java应用通过Hazelcast连接，其一莫名被终止

问题现象 #

两个Java应用通过Hazelcast组成集群，在Docker容器内初始连接正常，但不久后alpha-api.jar进程意外终止，导致Hazelcast节点失联，出现连接拒绝错误。本地直接运行启动脚本无此问题，调整JVM内存/CPU参数后仍未解决，怀疑是Docker配置相关问题。

相关配置代码 #

Hazelcast配置代码 #

final var clusterPort = 5701;
final var hazelcastConfig = new Config();
hazelcastConfig.setClusterName("alpha-cluster");
hazelcastConfig.setInstanceName(instanceName);
final var networkConfig = hazelcastConfig.getNetworkConfig();
networkConfig.setPort(5701);
final var joinConfig = networkConfig.getJoin();
joinConfig.getMulticastConfig().setEnabled(false);
final var tcpIpConfig = joinConfig.getTcpIpConfig();
tcpIpConfig.setEnabled(true);
for (int port = clusterPort; port < clusterPort + 10; port++) {
        tcpIpConfig.addMember(String.format("%s:%s", this.getHost(), port));
}
return new HazelcastClusterManager(hazelcastConfig);

Dockerfile #

FROM eclipse-temurin:latest
WORKDIR /usr/src/app
COPY alpha-api.jar ./alpha-api.jar
COPY alpha-admin-ui.jar ./alpha-admin-ui.jar
COPY start.sh ./start.sh
RUN chmod +x start.sh
EXPOSE 80
EXPOSE 8081
EXPOSE 5701
EXPOSE 5702
ENTRYPOINT [ "sh", "start.sh" ]

启动脚本start.sh #

#!/bin/sh
CPUS=2
MEMORY=512M
echo "Scripts starts ..."
java -Xmx$MEMORY -XX:ActiveProcessorCount=$CPUS -jar alpha-admin-ui.jar &
java -Xmx$MEMORY -XX:ActiveProcessorCount=$CPUS -jar alpha-api.jar &
wait

日志信息 #

初始正常连接日志 #

2023-10-09 09:38:41 Members {size:2, ver:2} [
2023-10-09 09:38:41     Member [localhost]:5701 - 0b43916b-caca-4f5b-afeb-a0fe696c9283 this
2023-10-09 09:38:41     Member [localhost]:5702 - cb43f74b-7cc2-4e53-9af2-9bdeff5823f7
2023-10-09 09:38:41 ]

异常日志 #

2023-10-09 09:38:43 INFO: [localhost]:5701 [vlab-alpha-cluster] [4.2.7] Could not connect to: localhost/127.0.0.1:5702. Reason: IOException[Connection refused to address localhost/127.0.0.1:5702]
2023-10-09 09:38:43 Oct 09, 2023 7:38:43 AM com.hazelcast.internal.server.tcp.TcpServerConnectionErrorHandler
2023-10-09 09:38:43 WARNING: [localhost]:5701 [vlab-alpha-cluster] [4.2.7] Removing connection to endpoint [localhost]:5702 Cause => java.io.IOException {Connection refused to address localhost/127.0.0.1:5702}, Error-Count: 5

排查与解决方案 #

1. 定位进程退出原因 #

修改启动脚本，将每个应用的输出重定向到单独日志文件，方便捕获崩溃细节：

#!/bin/sh
CPUS=2
MEMORY=512M
echo "Scripts starts ..."
java -Xmx$MEMORY -XX:ActiveProcessorCount=$CPUS -jar alpha-admin-ui.jar > admin-ui.log 2>&1 &
java -Xmx$MEMORY -XX:ActiveProcessorCount=$CPUS -jar alpha-api.jar > api.log 2>&1 &
wait

启动容器后，通过docker exec <容器名> cat /usr/src/app/api.log查看日志，确认是否存在OOM、未捕获异常等问题。

2. 调整Docker资源限制 #

Docker容器默认可能存在资源配额，导致JVM进程因资源不足被系统杀死。运行容器时显式指定资源：

docker run -d --name alpha-cluster -m 2g --cpus 2 -p 80:80 -p 8081:8081 -p 5701-5710:5701-5710 <你的镜像名>

通过docker stats alpha-cluster实时监控容器资源使用情况，确认是否达到限制阈值。

3. 修正Hazelcast网络配置 #

• 确保Hazelcast绑定到容器内所有网卡，避免仅绑定127.0.0.1导致的通信问题：

  networkConfig.setPort(5701);
  networkConfig.setPortAutoIncrement(true);
  networkConfig.getInterfaces().setEnabled(true).addInterface("0.0.0.0");
  

• 映射完整的端口范围：配置中添加了5701-5710的成员节点，Docker运行时需映射整个端口段，而非仅5701和5702。

4. 替换进程管理方式 #

原生shell脚本无法自动重启退出的进程，改用supervisord管理进程：

1. 修改Dockerfile安装supervisord：

   FROM eclipse-temurin:latest
   WORKDIR /usr/src/app
   RUN apt-get update && apt-get install -y supervisor
   COPY alpha-api.jar ./alpha-api.jar
   COPY alpha-admin-ui.jar ./alpha-admin-ui.jar
   COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf
   ENTRYPOINT ["/usr/bin/supervisord"]
   

2. 创建supervisord.conf配置文件：

   [supervisord]
   nodaemon=true
   
   [program:alpha-admin-ui]
   command=java -Xmx512M -XX:ActiveProcessorCount=2 -jar alpha-admin-ui.jar
   directory=/usr/src/app
   stdout_logfile=/usr/src/app/admin-ui.log
   stderr_logfile=/usr/src/app/admin-ui.err.log
   autorestart=true
   
   [program:alpha-api]
   command=java -Xmx512M -XX:ActiveProcessorCount=2 -jar alpha-api.jar
   directory=/usr/src/app
   stdout_logfile=/usr/src/app/api.log
   stderr_logfile=/usr/src/app/api.err.log
   autorestart=true
   

内容的提问来源于stack exchange，提问作者Qeychon