Cassandra 5.0压缩时FileHandle引用泄漏问题求助

问题场景 #

在Cassandra 5.0环境中维护一张包含map类型字段的时序数据表，执行压缩操作时出现以下内存泄漏错误日志，切换ZGC、G1GC垃圾回收器后问题仍未解决：

[2024-08-10 17:00:55,814] [STORAGE] [ERROR] [Reference-Reaper]  Ref.java:243 - LEAK DETECTED: a reference (class org.apache.cassandra.io.util.FileHandle$Cleanup@1350358023:/data1/pp-data/cassandra/data/pp/tm_asset_data-3fdee2c055e811efbbb9ade82e603b4a/da-3gik_0m9d_2zt682gly48a602kk0-bti-Data.db) to class org.apache.cassandra.io.util.FileHandle$Cleanup@1350358023:/data1/pp-data/cassandra/data/pp/tm_asset_data-3fdee2c055e811efbbb9ade82e603b4a/da-3gik_0m9d_2zt682gly48a602kk0-bti-Data.db was not released before the reference was garbage collected
[2024-08-10 17:00:55,814] [STORAGE] [ERROR] [Reference-Reaper]  Ref.java:243 - LEAK DETECTED: a reference (class org.apache.cassandra.io.util.FileHandle$Cleanup@1350358023:/data1/pp-data/cassandra/data/pp/tm_asset_data-3fdee2c055e811efbbb9ade82e603b4a/da-3gik_0m9d_2zt682gly48a602kk0-bti-Data.db) to class org.apache.cassandra.io.util.FileHandle$Cleanup@1350358023:/data1/pp-data/cassandra/data/pp/tm_asset_data-3fdee2c055e811efbbb9ade82e603b4a/da-3gik_0m9d_2zt682gly48a602kk0-bti-Data.db was not released before the reference was garbage collected
[2024-08-10 17:00:55,814] [STORAGE] [ERROR] [Reference-Reaper]  Ref.java:243 - LEAK DETECTED: a reference (class org.apache.cassandra.io.util.FileHandle$Cleanup@2078760468:/data1/pp-data/cassandra/data/pp/tm_asset_data-3fdee2c055e811efbbb9ade82e603b4a/da-3gik_0m9d_2zt682gly48a602kk0-bti-Partitions.db) to class org.apache.cassandra.io.util.FileHandle$Cleanup@2078760468:/data1/pp-data/cassandra/data/pp/tm_asset_data-3fdee2c055e811efbbb9ade82e603b4a/da-3gik_0m9d_2zt682gly48a602kk0-bti-Partitions.db was not released before the reference was garbage collected
[2024-08-10 17:00:55,814] [STORAGE] [ERROR] [Reference-Reaper]  Ref.java:243 - LEAK DETECTED: a reference (class org.apache.cassandra.io.util.FileHandle$Cleanup@2078760468:/data1/pp-data/cassandra/data/pp/tm_asset_data-3fdee2c055e811efbbb9ade82e603b4a/da-3gik_0m9d_2zt682gly48a602kk0-bti-Partitions.db) to class org.apache.cassandra.io.util.FileHandle$Cleanup@2078760468:/data1/pp-data/cassandra/data/pp/tm_asset_data-3fdee2c055e811efbbb9ade82e603b4a/da-3gik_0m9d_2zt682gly48a602kk0-bti-Partitions.db was not released before the reference was garbage collected
[2024-08-10 17:00:55,815] [STORAGE] [ERROR] [Reference-Reaper]  Ref.java:243 - LEAK DETECTED: a reference (class org.apache.cassandra.io.util.FileHandle$Cleanup@2078760468:/data1/pp-data/cassandra/data/pp/tm_asset_data-3fdee2c055e811efbbb9ade82e603b4a/da-3gik_0m9d_2zt682gly48a602kk0-bti-Partitions.db) to class org.apache.cassandra.io.util.FileHandle$Cleanup@2078760468:/data1/pp-data/cassandra/data/pp/tm_asset_data-3fdee2c055e811efbbb9ade82e603b4a/da-3gik_0m9d_2zt682gly48a602kk0-bti-Partitions.db was not released before the reference was garbage collected
[2024-08-10 17:00:55,815] [STORAGE] [ERROR] [Reference-Reaper]  Ref.java:243 - LEAK DETECTED: a reference (class org.apache.cassandra.io.util.FileHandle$Cleanup@2078760468:/data1/pp-data/cassandra/data/pp/tm_asset_data-3fdee2c055e811efbbb9ade82e603b4a/da-3gik_0m9d_2zt682gly48a602kk0-bti-Partitions.db) to class org.apache.cassandra.io.util.FileHandle$Cleanup@2078760468:/data1/pp-data/cassandra/data/pp/tm_asset_data-3fdee2c055e811efbbb9ade82e603b4a/da-3gik_0m9d_2zt682gly48a602kk0-bti-Partitions.db was not released before the reference was garbage collected
[2024-08-10 17:00:55,815] [STORAGE] [ERROR] [Reference-Reaper]  Ref.java:243 - LEAK DETECTED: a reference (class org.apache.cassandra.io.util.FileHandle$Cleanup@2081245557:/data1/pp-data/cassandra/data/pp/tm_asset_data-3fdee2c055e811efbbb9ade82e603b4a/da-3gik_0m9d_2zt682gly48a602kk0-bti-Rows.db) to class org.apache.cassandra.io.util.FileHandle$Cleanup@2081245557:/data1/pp-data/cassandra/data/pp/tm_asset_data-3fdee2c055e811efbbb9ade82e603b4a/da-3gik_0m9d_2zt682gly48a602kk0-bti-Rows.db was not released before the reference was garbage collected
[2024-08-10 17:00:55,815] [STORAGE] [ERROR] [Reference-Reaper]  Ref.java:243 - LEAK DETECTED: a reference (class org.apache.cassandra.io.util.FileHandle$Cleanup@2081245557:/data1/pp-data/cassandra/data/pp/tm_asset_data-3fdee2c055e811efbbb9ade82e603b4a/da-3gik_0m9d_2zt682gly48a602kk0-bti-Rows.db) to class org.apache.cassandra.io.util.FileHandle$Cleanup@2081245557:/data1/pp-data/cassandra/data/pp/tm_asset_data-3fdee2c055e811efbbb9ade82e603b4a/da-3gik_0m9d_2zt682gly48a602kk0-bti-Rows.db was not released before the reference was garbage collected
[2024-08-10 17:00:55,815] [STORAGE] [ERROR] [Reference-Reaper]  Ref.java:243 - LEAK DETECTED: a reference (class org.apache.cassandra.io.util.FileHandle$Cleanup@2081245557:/data1/pp-data/cassandra/data/pp/tm_asset_data-3fdee2c055e811efbbb9ade82e603b4a/da-3gik_0m9d_2zt682gly48a602kk0-bti-Rows.db) to class org.apache.cassandra.io.util.FileHandle$Cleanup@2081245557:/data1/pp-data/cassandra/data/pp/tm_asset_data-3fdee2c055e811efbbb9ade82e603b4a/da-3gik_0m9d_2zt682gly48a602kk0-bti-Rows.db was not released before the reference was garbage collected
[2024-08-10 17:00:55,815] [STORAGE] [ERROR] [Reference-Reaper]  Ref.java:243 - LEAK DETECTED: a reference (class org.apache.cassandra.io.util.FileHandle$Cleanup@2081245557:/data1/pp-data/cassandra/data/pp/tm_asset_data-3fdee2c055e811efbbb9ade82e603b4a/da-3gik_0m9d_2zt682gly48a602kk0-bti-Rows.db) to class org.apache.cassandra.io.util.FileHandle$Cleanup@2081245557:/data1/pp-data/cassandra/data/pp/tm_asset_data-3fdee2c055e811efbbb9ade82e603b4a/da-3gik_0m9d_2zt682gly48a602kk0-bti-Rows.db was not released before the reference was garbage collected

解决方案 #

1. 调整压缩策略与参数 #

• 降低压缩并发度：修改cassandra.yaml中的concurrent_compactors参数，设置为CPU核心数的1/2（例如4核机器设为2），减少同时运行的压缩任务数量，避免资源竞争导致FileHandle引用无法正常释放。
• 切换为Leveled压缩策略：针对目标表执行以下CQL语句，改用LeveledCompactionStrategy并减小SSTable大小，降低单次压缩的数据量：

  ALTER TABLE pp.tm_asset_data WITH compaction = {'class': 'LeveledCompactionStrategy', 'sstable_size_in_mb': 64};
  

• 限制压缩内存：在cassandra.yaml中设置compaction_memory_limit_in_mb为堆内存的1/4（例如堆内存8G时设为2048），防止压缩过程中内存溢出导致引用清理失败。

2. 针对map类型字段优化 #

• 拆分大map字段：如果单条记录的map包含大量键值对，考虑将其拆分为多个独立列或单独的表，减少压缩时单条记录的内存占用。
• 启用map分块存储：在表定义中添加map_chunk_size_kb参数（例如设置为32），将大map拆分为多个块存储，降低压缩时的内存负载：

  ALTER TABLE pp.tm_asset_data WITH options = {'map_chunk_size_kb': 32};
  

3. 修复Cassandra版本漏洞 #

• 升级到最新补丁版本：Cassandra 5.0的后续小版本（如5.0.3及以上）已修复部分FileHandle资源泄漏问题，直接升级到对应版本可解决根源问题。
• 临时关闭泄漏检测（应急方案）：修改cassandra.yaml中的reference_reaper_check_interval为0，关闭Reference-Reaper的泄漏检测日志，但此方法仅隐藏问题，不解决根源，不推荐长期使用。

4. JVM参数优化 #

• 调整堆内存大小：确保堆内存不超过物理内存的50%（例如16G物理内存设为8G堆内存），减少内存碎片化，帮助GC及时回收引用对象。
• 启用并行引用处理：在JVM启动参数中添加-XX:+ParallelRefProcEnabled，加速引用类型对象的垃圾回收，提升FileHandle Cleanup引用的清理效率。

内容的提问来源于stack exchange，提问作者이상부