基于Kotlin的Spring应用部署至Kubernetes后Hazelcast Map数据不同步问题排查

Alright, let's break down what's going wrong here and fix it step by step.

Core Issues #

Your problem stems from two critical missteps in how you're managing Hazelcast instances in your Spring application:

1. You’re manually creating Hazelcast instances instead of leveraging Spring Boot’s auto-configured one
   Your CacheClientImplHazelcast class spins up a new HazelcastInstance directly in its init block, completely ignoring the Hazelcast configuration you’ve defined in application.yaml. This means:

   • The Kubernetes DNS discovery settings you set up aren’t being applied to these manually created instances.
   • You’re likely running multiple Hazelcast instances per Pod (one from Spring’s auto-configuration, one from your custom code), which explains the "extra" cluster members you’re seeing.

2. Each cache client is tied to its own isolated Hazelcast instance
   Since every CacheClientImplHazelcast creates its own instance, even if they somehow joined the same cluster, your data operations are bound to that specific instance. This is why writes on one node don’t show up on others—each client is working with a separate (even if clustered) instance, so you’re not using the shared cluster-wide map correctly.

Answering Your Questions #

Question 1: Why 4 members for 3 Kubernetes instances?

You’ve got two Hazelcast instances running in at least one of your Pods (probably one from Spring Boot’s auto-config, one from your manual code). The log shows 10.4.2.32 has two ports (5701 and 5702), which confirms two instances in that single Pod. Add the two instances from the other two Pods, and you get 4 total members.

Question 2: Why 4 members for 2 local instances?

Same root cause—each local app process is running two Hazelcast instances (auto-configured + manual), so 2 processes × 2 instances = 4 cluster members.

Fix Steps #

Let’s rewrite your code and configuration to use Spring Boot’s proper Hazelcast integration:

1. Remove manual HazelcastInstance creation
   Inject the auto-configured HazelcastInstance into your cache client instead of creating it yourself. This ensures the instance uses your application.yaml settings (including Kubernetes discovery).

   @Component
   class CacheClientImplHazelcast(private val hz: HazelcastInstance) {
   
       // No init block needed—Spring injects the pre-configured instance
   
       fun getAllData(): List<MyDto> {
           val map: IMap<String, MyDto> = hz.getMap("my-map")
           return map.values.toList()
       }
   
       fun putData(key: String, myDto: MyDto) {
           val map: IMap<String, MyDto> = hz.getMap("my-map")
           map.put(key, myDto)
       }
   
       override fun clear() {
           val map: IMap<String, MyDto> = hz.getMap("my-map")
           map.clear()
       }
   }
   

2. Add your custom serializer to the auto-configured Hazelcast config
   To keep your custom serializer, create a HazelcastConfigCustomizer bean to modify Spring’s auto-generated config instead of building a new one from scratch:

   @Configuration
   class HazelcastConfig {
       @Bean
       fun hazelcastConfigCustomizer(): HazelcastConfigCustomizer {
           return HazelcastConfigCustomizer { config ->
               val serializer = SerializerConfig()
                   .setTypeClass(MyDto::class.java)
                   .setImplementation(MyDtoSerializer())
               config.serializationConfig.addSerializerConfig(serializer)
           }
       }
   }
   

3. Verify your Kubernetes Service configuration
   Your my-application-hs headless service looks correct for DNS discovery, but double-check that:

   • The selector component: my-application matches your Pods’ labels.
   • The Hazelcast port (5701) is correctly exposed in your Deployment’s container specs (ensure your Pods have port 5701 open).

Why This Works #

• Spring Boot will now create a single HazelcastInstance per Pod, using your application.yaml settings for Kubernetes DNS discovery.
• All cache operations will use this shared instance, so data will sync seamlessly across the entire cluster.
• You’ll see exactly 3 cluster members for 3 Kubernetes Pods (and 2 members for 2 local instances), matching your application instance count.

After making these changes, redeploy your app and check the logs—you should see the correct number of cluster members, and data should sync across all nodes.

内容的提问来源于stack exchange，提问作者Sjoerd222888