Let's break down your questions one by one, and dig into why your commitlog is growing unchecked even with your calculated memtable threshold.

1. What problems does persistent commitlog growth cause? #

A commitlog that's ballooning to 7.8GB (and still growing) poses several critical risks to your Cassandra cluster:

• Disk exhaustion: The most immediate threat—if the commitlog fills up your disk, the node will crash and stop accepting writes entirely. Cassandra requires free disk space to write new commitlog segments.
• Extended recovery time: If the node fails, Cassandra has to replay all unflushed commitlog entries to restore data consistency. Larger logs mean longer replay times, increasing cluster downtime.
• Degraded write performance: While commitlog writes are sequential, disk space pressure can trigger OS-level disk throttling or IO contention, which indirectly slows down write operations.
• Potential GC pressure (if using on-heap commitlogs): Though default commitlogs are off-heap, if you've configured on-heap storage, excessive commitlog size can eat into heap memory and trigger more frequent garbage collection pauses.

2. Does commitlog growth mean we haven't hit the memtable threshold? #

Not necessarily. Commitlog cleanup is tied to memtable flushes, but there are scenarios where memtables might reach their threshold without triggering commitlog pruning:

• Flush operations are blocked: If disk IO is saturated, SSTable writes can back up, causing memtable flush tasks to queue. Until the flush completes, the corresponding commitlog segments can't be marked for deletion.
• Commitlog archiving is enabled: If commitlog_archiving_enabled is turned on, old commitlog segments are retained until the archiving process finishes. Slow archiving (e.g., to remote storage) can prevent log cleanup.
• Misaligned threshold logic: Your calculated memtable_cleanup_threshold might not reflect the actual trigger condition Cassandra uses—we'll dive into this next.

3. Why isn't the memtable flush triggering when hitting the calculated 455MB threshold? #

First, let's clarify the formula you're using. The memtable_cleanup_threshold is calculated as:

memtable_cleanup_threshold = (memtable_total_space_in_mb) / (memtable_flush_writers + 1)

Where memtable_total_space_in_mb is the combined limit of memtable_heap_space_in_mb and memtable_offheap_space_in_mb (4GB total in your case). That gives you ~455MB, but here's why the flush might not be firing:

Possible root causes: #

• Actual memtable size hasn't hit the threshold: The 455MB is the total memtable size across all tables that triggers flushes. Use nodetool tablestats to sum up the Memtable Size for all tables—you might find the total is still below this value.
• Flush threads are overwhelmed: With memtable_flush_writers=8, you have 8 concurrent flush threads, but if all are busy flushing large memtables from other tables, new flush requests will queue. Check nodetool tpstats for pending tasks in the MemtableFlushWriter pool.
• Table-level memtable overrides: Individual tables might have custom memtable_heap_space_in_mb or memtable_offheap_space_in_mb settings that override the global threshold, preventing flushes from triggering when expected.
• Failed flush operations: If a memtable flush fails (e.g., due to disk errors, permission issues, or corrupted SSTables), Cassandra will retain the corresponding commitlog segment. Check your system.log for errors like Failed to flush memtable or SSTable write failed.
• Commitlog space limits aren't enforced: Since commitlog_total_space_in_mb is commented out, Cassandra uses the default behavior: it limits commitlog total size to 25% of the disk's free space (minimum 4GB, maximum 32GB). If your disk has plenty of free space, the commitlog will keep growing until memtables are flushed—so if flushes are blocked, the log will just keep expanding.

Troubleshooting steps to confirm: #

1. Run nodetool tablestats to verify total memtable size across all tables.
2. Use nodetool tpstats to check for pending tasks in the MemtableFlushWriter thread pool.
3. Inspect the system.log for flush-related errors or warnings.
4. Monitor disk IO with tools like iostat or dstat to check for bottlenecks.
5. Check if any tables have custom memtable configurations with DESCRIBE TABLE <keyspace>.<table>.

内容的提问来源于stack exchange，提问作者Coder