Spring+Hazelcast+K8s：多Pod部署时Hazelcast节点启动失败

背景 #

我正在为运行在Kubernetes中的Spring Boot API配置Hazelcast，该API部署在两个需要互相通信的Pod中。

配置信息 #

我定义了如下Hazelcast Java配置：

Maven依赖 #

<dependency>
  <groupId>com.hazelcast</groupId>
  <artifactId>hazelcast</artifactId>
  <version>5.2.2</version>
</dependency>

Java代码配置 #

Config config = getCommonConfig();
JoinConfig joinConfig = config.getNetworkConfig().getJoin();
joinConfig.getMulticastConfig().setEnabled(false);
config.getNetworkConfig().getJoin().getKubernetesConfig().setEnabled(true)
        .setProperty("namespace", namespace)
        .setProperty("service-name", service);

问题描述 #

当仅运行一个Pod时，API无错误正常运行。

但使用kubectl scale deployment <deployment_name> --replicas=2命令扩容应用后，第二个Pod启动失败，抛出如下堆栈跟踪：

WARN ConfigServletWebServerApplicationContext : Exception encountered during context initialization 
cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException:
Error creating bean with name 'hazelcastInstance' defined in class path resource [org/springframework/boot/autoconfigure/hazelcast/HazelcastServerConfiguration$HazelcastServerConfigConfiguration.class]:
Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException:
Failed to instantiate [com.hazelcast.core.HazelcastInstance]:
Factory method 'hazelcastInstance' threw exception; nested exception is java.lang.IllegalStateException: Node failed to start!

该堆栈信息模糊，将LOG_LEVEL设置为DEBUG也未获取到更多日志。

看起来，HazelcastServerConfiguration中定义的如下@Bean在双Pod运行时实例化失败，但单Pod时可正常运行：

@Configuration(
    proxyBeanMethods = false
)
@ConditionalOnSingleCandidate(Config.class)
static class HazelcastServerConfigConfiguration {
    HazelcastServerConfigConfiguration() {
    }

    @Bean
    HazelcastInstance hazelcastInstance(Config config) {
        return HazelcastServerConfiguration.getHazelcastInstance(config);
    }
}

已尝试的操作 #

• 在hazelcast依赖外添加hazelcast-kubernetes，问题依旧。
• 设置LOG_LEVEL=DEBUG以获取更多信息，但未新增日志内容。

Kubernetes配置 #

service.yaml #

apiVersion: v1
kind: Service
metadata:
  name: FOO
  [...]
spec:
  ports:
    - port: 8080
      protocol: TCP
      name: http
    - port: 5701
      protocol: TCP
      name: hazelcast
  type: ClusterIP
  selector:
    app.kubernetes.io/name: FOO
    app.kubernetes.io/instance: BAR

deployment.yaml #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: FOO
  labels:
    app.kubernetes.io/name: FOO
    [...]
spec:
  replicas: 2
  [...]
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
            - name: hazelcast
              containerPort: 5701
              protocol: TCP

内容的提问来源于stack exchange，提问作者AlbertCalmus