批量加载数据后Cassandra表无法查询，MmappedRegions抛出AssertionError

我在单节点环境运行Cassandra 5.0.2，通过Python程序向包含key(bigint)、value(text)字段的article_infos表加载数十GB文本数据。每次加载大量数据后，该表会无法查询，cqlsh执行查询时返回：

cqlsh> select * from article_infos limit 5;
ReadFailure: Error from server: code=1300 [Replica(s) failed to execute read] message="Operation failed - received 0 responses and 1 failures: UNKNOWN from /10.1.1.93:7000" info={'consistency': 'ONE', 'required_responses': 1, 'received_responses': 0, 'failures': 1, 'error_code_map': {'10.1.1.93': '0x0000'}}

同时Cassandra日志会打印如下错误：

ERROR [ReadStage-2] 2025-03-14 10:03:29,385 JVMStabilityInspector.java:70 - Exception in thread Thread[ReadStage-2,5,SharedPool]
java.lang.AssertionError: 7671074 > 7340032
at org.apache.cassandra.io.util.MmappedRegions$State.floor(MmappedRegions.java:363)
at org.apache.cassandra.io.util.MmappedRegions.floor(MmappedRegions.java:242)
at org.apache.cassandra.io.util.MmapRebufferer.rebuffer(MmapRebufferer.java:40)
at org.apache.cassandra.io.tries.Walker.<init>(Walker.java:75)
at org.apache.cassandra.io.tries.ValueIterator.<init>(ValueIterator.java:96)
at org.apache.cassandra.io.tries.ValueIterator.<init>(ValueIterator.java:80)
at org.apache.cassandra.io.sstable.format.bti.PartitionIndex$IndexPosIterator.<init>(PartitionIndex.java:407)
at org.apache.cassandra.io.sstable.format.bti.PartitionIterator.<init>(PartitionIterator.java:113)
at org.apache.cassandra.io.sstable.format.bti.PartitionIterator.create(PartitionIterator.java:75)
at org.apache.cassandra.io.sstable.format.bti.BtiTableReader.coveredKeysIterator(BtiTableReader.java:295)
at org.apache.cassandra.io.sstable.format.bti.BtiTableScanner$BtiScanningIterator.prepareToIterateRow(BtiTableScanner.java:114)
at org.apache.cassandra.io.sstable.format.SSTableScanner$BaseKeyScanningIterator.computeNext(SSTableScanner.java:264)
at org.apache.cassandra.io.sstable.format.SSTableScanner$BaseKeyScanningIterator.computeNext(SSTableScanner.java:244)
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at org.apache.cassandra.io.sstable.format.SSTableScanner.hasNext(SSTableScanner.java:206)
at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:90)
at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:375)
at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:187)
at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156)
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$4.hasNext(UnfilteredPartitionIterators.java:264)
at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:90)
at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:334)
at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:201)
at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:186)
at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:48)
at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:372)
at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:2210)
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2607)
at org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163)
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
ERROR [Reference-Reaper] 2025-03-14 10:03:55,267 Ref.java:243 - LEAK DETECTED: a reference (class org.apache.cassandra.io.util.FileHandle$Cleanup@1883489551:/scratch/USER/cassandra/data/PROJECT/article_infos-0e9822e0ff5111ef8c7267a1b8a131a2/da-3gok_1b58_3yj682gtxbd0q6ceck-bti-Data.db) to class org.apache.cassandra.io.util.FileHandle$Cleanup@1883489551:/scratch/USER/cassandra/data/PROJECT/article_infos-0e9822e0ff5111ef8c7267a1b8a131a2/da-3gok_1b58_3yj682gtxbd0q6ceck-bti-Data.db was not released before the reference was garbage collected
ERROR [Reference-Reaper] 2025-03-14 10:03:55,268 Ref.java:243 - LEAK DETECTED: a reference (class org.apache.cassandra.io.util.FileHandle$Cleanup@34244591:/scratch/USER/cassandra/data/PROJECT/article_infos-0e9822e0ff5111ef8c7267a1b8a131a2/da-3gok_1b58_3yj682gtxbd0q6ceck-bti-Partitions.db) to class org.apache.cassandra.io.util.FileHandle$Cleanup@34244591:/scratch/USER/cassandra/data/PROJECT/article_infos-0e9822e0ff5111ef8c7267a1b8a131a2/da-3gok_1b58_3yj682gtxbd0q6ceck-bti-Partitions.db was not released before the reference was garbage collected
ERROR [Reference-Reaper] 2025-03-14 10:03:55,268 Ref.java:243 - LEAK DETECTED: a reference (class org.apache.cassandra.io.util.FileHandle$Cleanup@713715721:/scratch/USER/cassandra/data/PROJECT/article_infos-0e9822e0ff5111ef8c7267a1b8a131a2/da-3gok_1b58_3yj682gtxbd0q6ceck-bti-Rows.db) to class org.apache.cassandra.io.util.FileHandle$Cleanup@713715721:/scratch/USER/cassandra/data/PROJECT/article_infos-0e9822e0ff5111ef8c7267a1b8a131a2/da-3gok_1b58_3yj682gtxbd0q6ceck-bti-Rows.db was not released before the reference was garbage collected

我当前使用Kafka及Datastax连接器写入数据，即使直接用INSERT语句写入也会出现相同问题。一段时间后错误会自动消失，表可正常查询。现咨询：

1. 该错误是否可以避免？
2. 大量数据写入后，如何判断数据库处于可安全查询的正常状态？

问题解答 #

1. 该错误是否可以避免？

这个错误是Cassandra 5.0.x版本中BTI（Binary Tree Index）格式SSTable的已知问题，根源是批量写入后Memtable flush生成的SSTable索引在内存映射时出现边界断言失败，同时伴随文件句柄泄漏。可通过以下方式避免：

• 降级SSTable格式：将表的SSTable格式从默认的bti改为legacy（旧BigFormat），执行ALTER TABLE语句：

  ALTER TABLE article_infos WITH storage_options = {'sstable_format': 'legacy'};
  

  重新写入数据后，新生成的SSTable会使用旧格式，规避BTI的索引断言问题。
• 升级Cassandra版本：后续稳定版本（如5.0.3及以上）大概率修复了该BTI相关的断言错误和文件句柄泄漏问题，升级到最新版可解决。
• 控制写入速率：避免短时间内写入过大流量，调整写入批次大小、降低并发数，让Memtable有足够时间flush，减少大SSTable生成概率，降低触发断言失败的风险。

2. 大量数据写入后，如何判断数据库处于可安全查询的正常状态？

可通过以下方式确认：

• 监控Memtable flush状态：执行nodetool tablestats article_infos命令，查看Pending flushes数值为0，且Memtable usage回到较低水平，说明内存中数据已全部flush到磁盘SSTable。
• 检查SSTable状态：执行nodetool listss tables -t article_infos，确认所有SSTable状态为NORMAL，无FLUSHING或其他异常状态的文件。
• 测试查询：先执行小范围查询（如SELECT * FROM article_infos LIMIT 10），若能正常返回结果，再尝试范围查询或全表扫描（单节点环境需注意全表扫描对性能的影响），确认无ReadFailure错误。
• 查看日志：检查Cassandra日志（默认路径logs/system.log），确认无新的ReadStage错误、断言失败或文件句柄泄漏日志，连续一段时间内日志无异常。
• 监控系统指标：确认节点CPU、内存、磁盘IO使用率恢复到正常水平，无持续高IO或内存占用情况，说明节点已完成写入后的后台处理（如flush、compaction）。

内容的提问来源于stack exchange，提问作者Hinton