Spring Boot 3.1.5升级后Infinispan集群节点数据交换失败排查

问题现象 #

集群中两个节点可互相发现，但节点间数据交换失败，抛出NullPointerException，核心错误栈指向AbstractBytesObjectInput.checkPosLength方法中this.bytes为null，完整错误日志如下：

org.infinispan.CLUSTER : ISPN000474: Error processing request 0@host-a-25374

java.lang.NullPointerException: Cannot read the array length because "this.bytes" is null
at org.infinispan.marshall.core.AbstractBytesObjectInput.checkPosLength(AbstractBytesObjectInput.java:260)
at org.infinispan.marshall.core.AbstractBytesObjectInput.readByte(AbstractBytesObjectInput.java:107)
at org.infinispan.marshall.core.AbstractBytesObjectInput.readUnsignedByte(AbstractBytesObjectInput.java:113)
at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:356)
at org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:191)
at org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:220)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1524)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1464)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.lambda$up$1(JGroupsTransport.java:1682)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1674)
at org.jgroups.JChannel.up(JChannel.java:749)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:939)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:161)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:319)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:319)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:859)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:498)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:722)
at org.jgroups.protocols.VERIFY_SUSPECT2.up(VERIFY_SUSPECT2.java:119)
at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:193)
at org.jgroups.protocols.FD_SOCK2.up(FD_SOCK2.java:202)
at org.jgroups.protocols.MERGE3.up(MERGE3.java:288)
at org.jgroups.protocols.Discovery.up(Discovery.java:314)
at org.jgroups.protocols.RED.up(RED.java:119)
at org.jgroups.protocols.TP.passBatchUp(TP.java:1210)
at org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.passBatchUp(MaxOneThreadPerSender.java:287)
at org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPool.java:157)
at org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.run(MaxOneThreadPerSender.java:276)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)

环境信息 #

• Spring Boot版本：3.1.5
• Infinispan依赖：org.infinispan:infinispan-spring-boot3-starter-embedded:14.0.20.Final
• 部署环境：Java 17 + Tomcat 10
• JGroups配置（UDP协议）：

<config xmlns="urn:org:jgroups"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups-5.2.xsd">
   
   <UDP bind_addr="${jgroups.bind.address,jgroups.udp.address:SITE_LOCAL}"
        bind_port="${jgroups.bind.port,jgroups.udp.port:0}"
        mcast_addr="${jgroups.udp.mcast_addr:201.0.0.1}"
        mcast_port="${jgroups.udp.mcast_port:46655}"
        tos="0"
        ucast_send_buf_size="1m"
        mcast_send_buf_size="1m"
        ucast_recv_buf_size="20m"
        mcast_recv_buf_size="25m"
        ip_ttl="${jgroups.ip_ttl:2}"
        thread_naming_pattern="pl"
        diag.enabled="${jgroups.diag.enabled:false}"
        bundler_type="transfer-queue"
        bundler.max_size="${jgroups.bundler.max_size:64000}"

        thread_pool.min_threads="${jgroups.thread_pool.min_threads:0}"
        thread_pool.max_threads="${jgroups.thread_pool.max_threads:200}"
        thread_pool.keep_alive_time="60000"

        thread_pool.thread_dumps_threshold="${jgroups.thread_dumps_threshold:10000}"
   />
   <RED/>
   <PING num_discovery_runs="3"/>
   <MERGE3 min_interval="10000"
           max_interval="30000"
   />
   <FD_SOCK2 offset="${jgroups.fd.port-offset:50000}"/>
   <FD_ALL3/>
   <VERIFY_SUSPECT2 timeout="1000"/>
   <pbcast.NAKACK2 xmit_interval="100"
                   xmit_table_num_rows="50"
                   xmit_table_msgs_per_row="1024"
                   xmit_table_max_compaction_time="30000"
                   resend_last_seqno="true"
   />
   <UNICAST3 xmit_interval="100"
             xmit_table_num_rows="50"
             xmit_table_msgs_per_row="1024"
             xmit_table_max_compaction_time="30000"
   />
   <pbcast.STABLE desired_avg_gossip="5000"
                  max_bytes="1M"
   />
   <pbcast.GMS print_local_addr="false"
               join_timeout="${jgroups.join_timeout:2000}"
   />
   <UFC max_credits="${jgroups.max_credits:4m}"
        min_threshold="0.40"
   />
   <MFC max_credits="${jgroups.max_credits:4m}"
        min_threshold="0.40"
   />
   <FRAG4 frag_size="${jgroups.frag_size:60000}"/>
</config>

• 差异现象：本地通过bootRun启动两台服务正常，部署到Tomcat后异常；Spring Boot 2.7.x版本无此问题，升级到3.1.5后触发。

排查方向与解决方案 #

1. 解决Tomcat类加载器冲突 #

Spring Boot 3.x切换为Jakarta EE API，独立Tomcat的类加载机制与内嵌Tomcat存在差异，易引发Marshalling相关类冲突：

• 检查Tomcat全局lib目录与应用WEB-INF/lib，确保仅应用包中包含Infinispan/JGroups依赖，删除Tomcat全局lib中的冲突包。
• 在Tomcat的context.xml中为应用添加配置：

<Context antiResourceLocking="true" antiJARLocking="true">
    <!-- 其他配置 -->
</Context>

2. 适配Spring Boot 3.x的Marshaller配置 #

显式指定GlobalMarshaller的类加载器，适配Jakarta EE环境：

@Bean
public GlobalMarshaller globalMarshaller() {
    GlobalMarshaller marshaller = new GlobalMarshaller();
    marshaller.setClassLoader(Thread.currentThread().getContextClassLoader());
    return marshaller;
}

同时检查自定义序列化类，确保其在Spring Boot 3.x下能被正确加载，无类路径变更导致的序列化失败。

3. 调整JGroups传输层配置 #

针对部署环境的网络特性，调整JGroups参数：

• 暂时禁用UDP消息打包，将bundler_type改为noop：

<UDP ... bundler_type="noop" ... />

• 对齐分片与打包大小，将FRAG4的frag_size调整为与UDP的bundler.max_size一致：

<FRAG4 frag_size="${jgroups.frag_size:64000}"/>

• 若多播存在问题，替换PING协议为TCPPING（指定静态节点列表）：

<TCPPING initial_hosts="host-a[port],host-b[port]" />

4. 验证版本兼容性 #

• 升级Infinispan到最新兼容版本（如14.0.21.Final及以上），修复已知的Spring Boot 3.x集成bug。
• 检查Spring Boot自动配置是否覆盖了Infinispan默认参数，避免配置变更引发的Marshalling异常。

5. 开启序列化调试日志 #

若以上方案无效，启用调试日志定位问题对象：

logging.level.org.infinispan.marshall=DEBUG
logging.level.org.jgroups=DEBUG

分析日志中序列化的对象，确认是否存在无法序列化的对象或空值传递异常。

内容的提问来源于stack exchange，提问作者kgg