Hazelcast 4.2.5 ReplicatedMap频繁出现ClassCastException问题求助

我们在Kubernetes环境的Tomcat上部署了Web应用，使用Hazelcast 4.2.5版本，应用日志中频繁（每5秒一次）出现带有堆栈跟踪的ClassCastException。

异常信息 #

java.lang.ClassCastException: class java.lang.String cannot be cast to class com.hazelcast.internal.serialization.impl.HeapData (java.lang.String is in module java.base of loader 'bootstrap'; com.hazelcast.internal.serialization.impl.HeapData is in unnamed module of loader org.apache.catalina.loader.ParallelWebappClassLoader @2f04993d)
27-Oct-2022 22:57:56.357 WARNING [hz.rogueUsers.cached.thread-2] com.hazelcast.internal.metrics.impl.MetricsCollectionCycle.null Collecting metrics from source com.hazelcast.replicatedmap.impl.ReplicatedMapService failed
at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
at java.base/java.lang.Thread.run(Thread.java:834)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217)
at com.hazelcast.spi.impl.executionservice.impl.DelegateAndSkipOnConcurrentExecutionDecorator$DelegateDecorator.run(DelegateAndSkipOnConcurrentExecutionDecorator.java:77)
at com.hazelcast.internal.metrics.impl.MetricsService.collectMetrics(MetricsService.java:154)
at com.hazelcast.internal.metrics.impl.MetricsService.collectMetrics(MetricsService.java:160)
at com.hazelcast.internal.metrics.impl.MetricsRegistryImpl.collect(MetricsRegistryImpl.java:316)
at com.hazelcast.internal.metrics.impl.MetricsCollectionCycle.collectDynamicMetrics(MetricsCollectionCycle.java:88)
at com.hazelcast.replicatedmap.impl.ReplicatedMapService.provideDynamicMetrics(ReplicatedMapService.java:387)
at com.hazelcast.replicatedmap.impl.ReplicatedMapService.getStats(ReplicatedMapService.java:357)
at com.hazelcast.replicatedmap.impl.ReplicatedMapService.getLocalReplicatedMapStats(ReplicatedMapService.java:197)
at com.hazelcast.replicatedmap.impl.LocalReplicatedMapStatsProvider.getLocalReplicatedMapStats(LocalReplicatedMapStatsProvider.java:85)

当前Hazelcast配置代码 #

private static HazelcastInstance setupHazelcastConfig() {
    Config config = new Config();
    config.setInstanceName("rogueUsers");
    NetworkConfig network = config.getNetworkConfig();
    network.setPort(5701).setPortCount(20);
    network.setPortAutoIncrement(true);
    JoinConfig join = network.getJoin();
    join.getMulticastConfig().setEnabled(true);
//        join.getTcpIpConfig()
//        .setEnabled(true);

    HazelcastInstance hz = Hazelcast.getOrCreateHazelcastInstance(config);

    ReplicatedMapConfig replicatedMapConfig =
            config.getReplicatedMapConfig("rogueUsers");

    replicatedMapConfig.setInMemoryFormat(InMemoryFormat.BINARY);
    replicatedMapConfig.setAsyncFillup(true);
    replicatedMapConfig.setStatisticsEnabled(true);
    replicatedMapConfig.setSplitBrainProtectionName("splitbrainprotection-name");

    ReplicatedMap<String, String> map = hz.getReplicatedMap("rogueUsers");
    map.addEntryListener(new RogueEntryListener());

    return hz;
}

这个问题是配置时机错误导致的：你在创建Hazelcast实例之后才修改ReplicatedMapConfig，此时实例已经基于默认配置初始化完成，后续的配置修改无法生效，进而引发序列化相关的类型转换异常（因为ReplicatedMap实际使用的是默认内存格式而非你设置的BINARY格式）。

具体修复步骤：

1. 调整配置顺序：在调用Hazelcast.getOrCreateHazelcastInstance(config)创建实例之前，完成所有ReplicatedMapConfig的自定义配置。
2. 确保配置生效：所有针对Map的配置修改必须在实例初始化前完成，这样配置才能被应用到实例中。

修改后的代码如下：

private static HazelcastInstance setupHazelcastConfig() {
    Config config = new Config();
    config.setInstanceName("rogueUsers");
    NetworkConfig network = config.getNetworkConfig();
    network.setPort(5701).setPortCount(20);
    network.setPortAutoIncrement(true);
    JoinConfig join = network.getJoin();
    join.getMulticastConfig().setEnabled(true);
//        join.getTcpIpConfig()
//        .setEnabled(true);

    // 在创建实例前配置ReplicatedMap
    ReplicatedMapConfig replicatedMapConfig =
            config.getReplicatedMapConfig("rogueUsers");

    replicatedMapConfig.setInMemoryFormat(InMemoryFormat.BINARY);
    replicatedMapConfig.setAsyncFillup(true);
    replicatedMapConfig.setStatisticsEnabled(true);
    replicatedMapConfig.setSplitBrainProtectionName("splitbrainprotection-name");

    // 现在创建Hazelcast实例
    HazelcastInstance hz = Hazelcast.getOrCreateHazelcastInstance(config);

    ReplicatedMap<String, String> map = hz.getReplicatedMap("rogueUsers");
    map.addEntryListener(new RogueEntryListener());

    return hz;
}

额外注意事项：

• 在Kubernetes环境中，Multicast通常无法正常工作，建议启用TcpIpConfig并配置集群内的节点发现（比如基于Kubernetes DNS或服务发现），避免集群节点无法互相发现引发的其他问题。
• 若修改配置后仍有异常，可尝试临时禁用ReplicatedMap的统计（setStatisticsEnabled(false)），验证是否是统计收集流程的序列化问题，但优先保证配置时机正确。

内容的提问来源于stack exchange，提问作者anjanb