Ignite节点出现SIGSEGV错误的原因排查请求

我们拥有一个包含10个节点的Ignite集群（每个节点配置12核CPU、62GB内存）。部分节点会无预警故障，查看Ignite日志未发现任何错误、警告或说明。首个节点故障后不久，第二个节点也会故障，随后出现分区丢失错误。经排查发现，故障节点在服务器日志中收到SIGSEGV信号，相关日志如下：

Apr 19 07:12:25 tr-ignite-6 service.sh[478792]: # SIGSEGV (0xb) at pc=0x00007f97110526d0, pid=478876, tid=0x00007f9675dc6700
Apr 19 08:02:22 tr-ignite-9 service.sh[494185]: # SIGSEGV (0xb) at pc=0x00007fbe71052854, pid=494269, tid=0x00007fbdd9f0e700
Apr 19 08:02:35 tr-ignite-12 service.sh[387860]: # SIGSEGV (0xb) at pc=0x00007f2501052840, pid=387944, tid=0x00007f2475e77700

Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: # A fatal error has been detected by the Java Runtime Environment:
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: #
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: #  SIGSEGV (0xb) at pc=0x00007efccc77e6ee, pid=1607467, tid=0x00007efc4c43b700
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: #
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: # JRE version: Java(TM) SE Runtime Environment (8.0_281-b09) (build 1.8.0_281-b09)
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.281-b09 mixed mode linux-amd64 compressed oops)
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: # Problematic frame:
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: # v  ~StubRoutines::jbyte_disjoint_arraycopy
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: #
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: # Core dump written. Default location: /var/lib/apache-ignite/core or core.1607467
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: #
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: # An error report file with more information is saved as:
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: # /var/lib/apache-ignite/hs_err_pid1607467.log
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: Compiled method (c2) 1403659804 15609       4       org.apache.ignite.internal.binary.BinaryWriterExImpl::write (13 bytes)
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: total in heap  [0x00007efccf1c29d0,0x00007efccf1c3138] = 1896
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: relocation     [0x00007efccf1c2af8,0x00007efccf1c2b40] = 72
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: main code      [0x00007efccf1c2b40,0x00007efccf1c2e20] = 736
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: stub code      [0x00007efccf1c2e20,0x00007efccf1c2e58] = 56
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: oops           [0x00007efccf1c2e58,0x00007efccf1c2e60] = 8
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: metadata       [0x00007efccf1c2e60,0x00007efccf1c2ea8] = 72
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: scopes data    [0x00007efccf1c2ea8,0x00007efccf1c3038] = 400
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: scopes pcs     [0x00007efccf1c3038,0x00007efccf1c30d8] = 160
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: dependencies   [0x00007efccf1c30d8,0x00007efccf1c30e8] = 16
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: handler table  [0x00007efccf1c30e8,0x00007efccf1c3118] = 48
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: nul chk table  [0x00007efccf1c3118,0x00007efccf1c3138] = 32
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: #
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: # If you would like to submit a bug report, please visit:
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: #   http://bugreport.java.com/bugreport/crash.jsp
Mar 24 08:28:02 tr-ignite-2 service.sh[1607384]: #

可能的原因分析 #

• JVM版本固有缺陷：日志显示使用JDK 8u281，该版本HotSpot VM的StubRoutines::jbyte_disjoint_arraycopy数组拷贝存根例程存在已知崩溃问题，高并发场景下内存访问逻辑异常会触发SIGSEGV。
• 物理内存硬件故障：SIGSEGV是非法内存访问的直接表现，可能是节点服务器的内存模块损坏，导致JVM执行内存拷贝时访问了无效内存地址。可通过内存检测工具排查硬件问题。
• Ignite二进制序列化异常：崩溃关联到org.apache.ignite.internal.binary.BinaryWriterExImpl::write方法，说明故障发生在二进制数据写入环节。可能是自定义序列化逻辑存在问题，或是Ignite二进制协议在特定数据场景下触发了JVM内存访问错误。
• 并发内存竞争破坏：Ignite是高并发分布式系统，多线程同时操作共享内存区域（如二进制缓冲区）时，可能导致内存结构损坏，进而引发JVM内存访问异常。
• 系统资源配置冲突：尽管节点配置62GB内存，但如果系统层面存在cgroup内存限制、swap配置不合理，或是其他进程抢占内存资源，可能导致JVM内存分配异常，间接触发SIGSEGV。

内容的提问来源于stack exchange，提问作者Ayşe Y