Cassandra 4.0.8节点数据清理时Java崩溃问题求助

问题背景 #

• 运行环境：CentOS Stream release 8，Cassandra 4.0.8，Java 11.0.17（HotSpot 64-Bit Server VM 11.0.17+10-LTS-269，CMS GC）
• 触发场景：节点bootstrap完成后执行清理操作，在大量数据流式传输、压缩等高负载场景下偶发节点崩溃，无法稳定复现
• 崩溃核心特征：JVM触发致命错误，信号为SIGBUS (0x7)（BUS_ADRERR，内存地址访问错误），错误发生在~StubRoutines::updateBytesCRC32，当前线程为CompactionExecutor:8

核心转储提取信息 #

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007fe388431daa, pid=3006650, tid=3008039
#
# JRE version: Java(TM) SE Runtime Environment 18.9 (11.0.17+10) (build 11.0.17+10-LTS-269)
# Java VM: Java HotSpot(TM) 64-Bit Server VM 18.9 (11.0.17+10-LTS-269, mixed mode, tiered, compressed oops, concurrent mark sweep gc, linux-amd64)
# Problematic frame:
# v  ~StubRoutines::updateBytesCRC32
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e" (or dumping to /home/cassandra/core.3006650)
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#
---------------  S U M M A R Y ------------

Command Line: -ea -da:net.openhft... -XX:+UseThreadPriorities -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:+AlwaysPreTouch -XX:-UseBiasedLocking -XX:+UseTLAB -XX:+ResizeTLAB -XX:+UseNUMA -XX:+PerfDisableSharedMem -Djava.net.preferIPv4Stack=true -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSWaitDuration=10000 -XX:+CMSParallelInitialMarkEnabled -XX:+CMSEdenChunksRecordAlways -XX:+CMSClassUnloadingEnabled -Djdk.attach.allowAttachSelf=true --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=java.management.rmi/com.sun.jmx.remote.internal.rmi=ALL-UNNAMED --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED --add-exports=java.rmi/sun.rmi.server=ALL-UNNAMED --add-exports=java.sql/java.sql=ALL-UNNAMED --add-opens=java.base/java.lang.module=ALL-UNNAMED --add-opens=java.base/jdk.internal.loader=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/jdk.internal.reflect=ALL-UNNAMED --add-opens=java.base/jdk.internal.math=ALL-UNNAMED --add-opens=java.base/jdk.internal.module=ALL-UNNAMED --add-opens=java.base/jdk.internal.util.jar=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -Dio.netty.tryReflectionSetAccessible=true -Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/home/cassandra/apache-cassandra/logs/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=10485760 -Xms7961M -Xmx7961M -Xmn400M -XX:+UseCondCardMark -XX:CompileCommandFile=/home/cassandra/apache-cassandra/conf/hotspot_compiler -javaagent:/home/cassandra/apache-cassandra/lib/jamm-0.3.2.jar -Dcassandra.jmx.remote.port=7199 -Dcom.sun.management.jmxremote.rmi.port=7199 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password -Djava.library.path=/home/cassandra/apache-cassandra/lib/sigar-bin -Dcassandra.libjemalloc=/usr/lib64/libjemalloc.so.2 -XX:OnOutOfMemoryError=kill -9 %p -Dlogback.configurationFile=logback.xml -Dcassandra.logdir=/home/cassandra/apache-cassandra/logs -Dcassandra.storagedir=/home/cassandra/apache-cassandra/data org.apache.cassandra.service.CassandraDaemon

Host: Intel(R) Core(TM) i5-6260U CPU @ 1.80GHz, 4 cores, 31G, CentOS Stream release 8
Time: Tue Mar 14 15:01:44 2023 CET elapsed time: 2430.690820 seconds (0d 0h 40m 30s)

---------------  T H R E A D  ---------------

Current thread (0x00007f2b75aaee00):  JavaThread "CompactionExecutor:8" daemon [_thread_in_Java, id=3008039, stack(0x00007f25d5ddf000,0x00007f25d5e20000)]

Stack: [0x00007f25d5ddf000,0x00007f25d5e20000],  sp=0x00007f25d5e1c6f0,  free space=245k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
v  ~StubRoutines::updateBytesCRC32

siginfo: si_signo: 7 (SIGBUS), si_code: 2 (BUS_ADRERR), si_addr: 0x00007f4556a2d000

Register to memory mapping:

RAX=0x00000000d8464a00 is an unknown value
RBX=0x000000061f4c1c90 is an oop: java.nio.DirectByteBufferR
{0x000000061f4c1c90} - klass: 'java/nio/DirectByteBufferR'
RCX=0x00007fe3a138e9c0: <offset 0x000000000146f9c0> in /usr/jdk-11.0.17/lib/server/libjvm.so at 0x00007fe39ff1f000
RDX=0x0000000000000082 is an unknown value
RSP=0x00007f25d5e1c6f0 is pointing into the stack for thread: 0x00007f2b75aaee00
RBP=0x00007f25d5e1c6f0 is pointing into the stack for thread: 0x00007f2b75aaee00
RSI=0x00007f4556a2cfd0 points into unknown readable memory: 0x4f1f18b0cedc8a28 | 28 8a dc ce b0 18 1f 4f
RDI=0x00000000d18d1b56 is an unknown value
R8 =0x0000000029edee07 is an unknown value
R9 =0x0000000000000a19 is an unknown value

解决建议 #

1. 硬件层面排查 #

• 检查物理内存健康：使用memtest86+工具扫描内存，排除硬件故障导致的内存访问错误
• 检查存储设备状态：通过smartctl查看磁盘/SSD的SMART信息，确认是否存在坏块、IO超时等异常

2. JVM配置调整 #

• 切换GC算法：替换当前CMS GC为Cassandra 4.x推荐的G1GC，修改JVM启动参数：

  -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:ParallelGCThreads=4 -XX:ConcGCThreads=2
  

• 禁用DirectByteBuffer：添加JVM参数-Dcassandra.use_direct_buffers=false，强制使用堆内缓冲区，避免DirectByteBuffer释放后的野指针访问问题
• 调整堆内存参数：当前-Xmn400M偏小，可尝试调大新生代内存（如-Xmn2G），减少Minor GC频率，降低高负载下的内存波动

3. Cassandra配置优化 #

• 降低Compaction并发度：在cassandra.yaml中修改concurrent_compactors为CPU核心数的一半（如4核节点设为2），减少压缩阶段的资源竞争
• 调整清理操作时机：避免在高负载时段执行nodetool cleanup，选择业务低峰期操作，或者分批清理Keyspace/Table
• 升级Cassandra版本：升级到4.0系列的最新稳定版本（如4.0.11及以上），社区可能已修复压缩、清理阶段的内存访问Bug

4. Java版本升级 #

• 升级到Java 11的最新补丁版本（如11.0.21），该版本修复了多个与StubRoutines、内存访问相关的已知问题

5. 监控与日志增强 #

• 开启Cassandra的Compaction详细日志：修改logback.xml，将org.apache.cassandra.db.compaction的日志级别设为DEBUG
• 增加系统层面监控：通过vmstat、iostat、sar持续监控内存、磁盘IO、CPU使用率，当问题复现时收集完整的核心转储与日志，便于根因定位

内容的提问来源于stack exchange，提问作者BuckBazooka