Cassandra 3.11.9集群G1老年代GC长时间停顿问题求助

集群现状与核心问题 #

• 51节点Cassandra 3.11.9集群，单节点数据负载达600-800GB
• G1老年代GC出现超长停顿，最长达257秒，触发客户端（一致性级别为ONE）延迟峰值，严重影响应用响应

GC日志示例 #

28 | CHANGED | rc=0 >>
INFO  [Service Thread] 2023-07-14 01:56:45,583 GCInspector.java:285 - G1 Old Generation GC in 12039ms.  G1 Old Gen: 54648149088 -> 12146746552; G1 Survivor Space: 134217728 -> 0;
INFO  [Service Thread] 2023-07-14 02:14:24,394 GCInspector.java:285 - G1 Old Generation GC in 57918ms.  G1 Old Gen: 67780410096 -> 59704216816; Metaspace: 61436792 -> 61302392
INFO  [Service Thread] 2023-07-14 02:15:44,506 GCInspector.java:285 - G1 Old Generation GC in 64576ms.  G1 Old Gen: 67971190408 -> 64736391536;
INFO  [Service Thread] 2023-07-14 02:17:06,520 GCInspector.java:285 - G1 Old Generation GC in 66242ms.  G1 Old Gen: 68043573704 -> 66792790424;
INFO  [Service Thread] 2023-07-14 02:21:31,210 GCInspector.java:285 - G1 Old Generation GC in 257268ms.  G1 Old Gen: 68046631936 -> 67703054448;

254 | CHANGED | rc=0 >>
INFO  [Service Thread] 2023-07-14 02:36:26,170 GCInspector.java:285 - G1 Old Generation GC in 46654ms.  G1 Old Gen: 133621345752 -> 49403423024; Metaspace: 67436096 -> 67339688
INFO  [Service Thread] 2023-07-14 02:38:58,627 GCInspector.java:285 - G1 Old Generation GC in 89392ms.  G1 Old Gen: 133594285096 -> 103157948104;
INFO  [Service Thread] 2023-07-14 02:40:59,754 GCInspector.java:285 - G1 Old Generation GC in 93345ms.  G1 Old Gen: 135071359720 -> 105377369048; G1 Survivor Space: 33554432 -> 0;
INFO  [Service Thread] 2023-07-14 02:43:29,171 GCInspector.java:285 - G1 Old Generation GC in 106174ms.  G1 Old Gen: 133812654600 -> 119264140552; G1 Survivor Space: 234881024 -> 0;
INFO  [Service Thread] 2023-07-14 02:45:36,900 GCInspector.java:285 - G1 Old Generation GC in 95625ms.  G1 Old Gen: 135225564784 -> 99943593104;
INFO  [Service Thread] 2023-07-14 02:46:53,820 GCInspector.java:285 - G1 Old Generation GC in 55875ms.  G1 Old Gen: 133359614104 -> 60924511688; G1 Survivor Space: 872415232 -> 0;
INFO  [Service Thread] 2023-07-14 02:48:22,803 GCInspector.java:285 - G1 Old Generation GC in 38493ms.  G1 Old Gen: 133978126912 -> 36277631424;
INFO  [Service Thread] 2023-07-14 02:50:11,320 GCInspector.java:285 - G1 Old Generation GC in 34789ms.  G1 Old Gen: 134004918888 -> 35377344368;

250 | CHANGED | rc=0 >>
INFO  [Service Thread] 2023-07-14 00:18:52,262 GCInspector.java:285 - G1 Old Generation GC in 96017ms.  G1 Old Gen: 73628910144 -> 59159105432; Metaspace: 58018496 -> 57907432
INFO  [Service Thread] 2023-07-14 00:46:41,400 GCInspector.java:285 - G1 Old Generation GC in 30177ms.  G1 Old Gen: 41448088568 -> 24094354384; G1 Survivor Space: 67108864 -> 0;
INFO  [Service Thread] 2023-07-14 02:18:34,910 GCInspector.java:285 - G1 Old Generation GC in 40940ms.  G1 Old Gen: 74016882928 -> 27759131352; Metaspace: 57315192 -> 57128720
INFO  [Service Thread] 2023-07-14 02:36:02,256 GCInspector.java:285 - G1 Old Generation GC in 57658ms.  G1 Old Gen: 73488401080 -> 40838191112; Metaspace: 54701984 -> 54651552
INFO  [Service Thread] 2023-07-14 02:37:47,374 GCInspector.java:285 - G1 Old Generation GC in 87036ms.  G1 Old Gen: 73498188264 -> 65920831896;
INFO  [Service Thread] 2023-07-14 02:39:58,921 GCInspector.java:285 - G1 Old Generation GC in 111435ms.  G1 Old Gen: 73496794000 -> 70079092144;

已尝试方案 #

• 升级实例规格（增大JVM内存）：仅延迟问题出现时间，未根本解决
• 移除问题节点：新节点同步数据后很快出现同样GC问题
• 更换JVM及GC策略：试过官方JDK的G1GC、AZUL Zing等不同版本，无明显改善

当前限制条件 #

• 无法修改数据模型或集群架构（需跨团队协调，暂不可行）
• 无法升级Cassandra至4.x版本（需先完成应用适配，排期滞后）
• 临时缓解方案仅能对问题节点执行disablebinary，但会导致该节点无法提供服务，非长期合理方案

当前JVM参数 #

INFO  [main] 2023-07-17 18:40:11,668 CassandraDaemon.java:507 - JVM Arguments: [-javaagent:/opt/simility/include/exporters/jmxexporter/jmx_prometheus_javaagent-0.12.0.jar=7070:/opt/simility/include/exporters/jmxexporter/cassandra.yml, -ea, -javaagent:/opt/simility/include/cassandra/lib/jamm-0.3.0.jar, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -Xms64G, -Xmx64G, -XX:+ExitOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+UseG1GC, -XX:G1RSetUpdatingPauseTimePercent=5, -XX:MaxGCPauseMillis=500, -Djava.net.preferIPv4Stack=true, -Dcassandra.config=file:///opt/simility/conf/cassandra/cassandra.yaml, -Djava.rmi.server.hostname=172.33.135.28, -Dcom.sun.management.jmxremote.port=7199, -Dcom.sun.management.jmxremote.rmi.port=7199, -Dcom.sun.management.jmxremote.ssl=false, -Dcom.sun.management.jmxremote.authenticate=false, -Dcassandra.libjemalloc=/lib64/libjemalloc.so.1, -Dlogback.configurationFile=logback.xml, -Dcassandra.config=file:///opt/simility/conf/cassandra/cassandra.yaml, -Dcassandra.logdir=/opt/simility/log/cassandra, -Dcassandra.storagedir=/opt/simility/include/cassandra/data]

可行解决思路 #

1. G1GC参数针对性调优 #

• 调整停顿时间目标：当前-XX:MaxGCPauseMillis=500过于激进，实际GC远超该值，建议调高至1000-2000ms，避免G1为追求小停顿触发频繁混合GC或Full GC
• 增大Region大小：64G堆内存默认Region为16M，可设置-XX:G1HeapRegionSize=32M或64M，减少Region数量，降低RSet维护的CPU和内存开销
• 优化年轻代占比：添加-XX:G1NewSizePercent=20 -XX:G1MaxNewSizePercent=40，扩大年轻代范围，让更多对象在YGC阶段回收，减缓老年代增长速度
• 控制混合GC次数：设置-XX:G1MixedGCCountTarget=8，限制混合GC执行次数，避免老年代碎片持续积累
• 关闭自适应大小策略：添加-XX:-UseAdaptiveSizePolicy，手动固定内存区域参数，避免JVM动态调整导致的波动
• 增加预留内存：设置-XX:G1ReservePercent=15，为老年代预留更多内存空间，防止因内存不足触发Full GC

2. Cassandra配置优化 #

• 调整并发线程数：根据CPU核心数降低concurrent_reads和concurrent_writes，比如32核CPU设置concurrent_reads: 32、concurrent_writes: 16，减少线程竞争导致的内存分配压力
• 优化memtable刷盘队列：设置memtable_flush_queue_size: 8，避免memtable刷盘阻塞导致内存堆积
• 调整commitlog大小：将commitlog_segment_size_in_mb从32M调整为64M，减少commitlog切换频率，降低IO波动
• 优化缓存策略：检查key cache和row cache占用，若过高则适当降低key_cache_size_in_mb和row_cache_size_in_mb，避免挤占堆内存
• 增强GC日志监控：添加-Dcassandra.gc_log_threshold=100，记录所有超过100ms的GC事件，便于定位触发GC的业务场景

3. 系统层面优化 #

• 排查磁盘IO瓶颈：用iostat持续监控磁盘读写延迟，确保使用SSD且IO负载在合理范围，磁盘性能不足会导致memtable刷盘缓慢，加剧内存堆积
• 禁用swap分区：Cassandra对swap极度敏感，执行swapoff -a并修改/etc/fstab永久禁用swap，避免JVM使用swap导致GC停顿暴涨
• 调整系统内存分配：确保系统预留足够空闲内存给操作系统缓存，避免JVM内存与系统缓存竞争
• 优化jemalloc配置：当前已启用jemalloc，可升级至最新稳定版，或添加环境变量MALLOC_CONF="background_thread:true,metadata_thp:auto"优化内存分配效率

4. 内存泄漏与大对象排查 #

• 定期分析堆内存：执行jmap -histo:live <pid>，对比多次结果找出持续增长的大对象或高频创建对象
• 启用JFR录制：添加JVM参数-XX:+UnlockCommercialFeatures -XX:+FlightRecorder，在GC停顿期间录制飞行快照，分析内存分配热点
• 检查大字段使用：排查列族中是否存在Blob等大字段，若存在则调整对应列族的缓存策略（如禁用row cache），减少大对象在堆内存中的停留时间

内容的提问来源于stack exchange，提问作者Gabriel Dias