Cassandra集群部分节点突发100%CPU占用问题排查求助

Hey there, let's break down how to diagnose those random 100% CPU spikes on your Cassandra nodes. You've already got key data (tpstats, GC logs, table stats) which is perfect—here's a structured approach to dig into the root cause:

1. Start with GC Logs to Rule Out JVM Overhead #

Cassandra 2.2.x relies heavily on the JVM, so GC activity is often the first culprit for CPU spikes:

• Check Full GC frequency/duration: If Full GCs are firing often (more than once every few hours) or taking several seconds to complete, CMS (the default collector for 2.2.x) will hog CPU threads. For 64GB nodes, make sure your heap size isn't set over 32GB—going beyond that disables pointer compression, making GC exponentially more expensive.
• Analyze Young GC patterns: Frequent Young GCs with long pause times mean your new generation heap is sized incorrectly, or your workload is generating tons of short-lived objects (like temporary query results).
• Count active GC threads: CMS spawns GC threads based on CPU core count—if these threads are saturating all 8 cores, your application threads will starve for resources.

2. Parse tpstats to Identify Thread Pool Bottlenecks #

tpstats reveals exactly which Cassandra tasks are consuming CPU. Focus on these pools:

• ReadStage/RequestResponseStage: If pending queues are growing and active threads hit the core count limit, the node is swamped with read requests that are slow to process (e.g., large partition scans, high consistency level queries).
• MutationStage: Sudden write spikes, or writes involving massive column updates/TTL processing, can push this pool to max capacity.
• CompactionExecutor: Even if no pending compactions show up, check for active compactions—sometimes a compaction starts before it appears in the pending list, or tombstone cleanup tasks (which run in this pool) can drain CPU.
• InternalResponseStage: This handles inter-node communication (gossip, repairs, data sync). Frequent node flapping or ongoing repair jobs can overload this pool and spike CPU.

3. Use Table Stats to Spot Hotspots or Query Inefficiencies #

Compare your high-load multi-table stats with the post-load abc table stats to find anomalies:

• Sudden traffic spikes: Look for tables with a massive jump in read_count or write_count during the CPU spike. If a single partition gets hammered (thanks to Cassandra's hash-based partition distribution), the node owning that partition will get crushed.
• Large partitions: If any table has average partitions over 100MB, queries against those partitions force the node to serialize/deserialize huge datasets, eating up CPU.
• Tombstone overload: Too many tombstones (from frequent deletes or expired TTLS) force the node to filter thousands of dead rows during queries—this is a common hidden CPU drain, especially for range queries.

4. Check for Cluster-Wide Maintenance or Client Issues #

Random spikes often tie to periodic or unexpected operations:

• Repair jobs: Incremental or full repairs trigger cross-node data validation and transfer, which can CPU-bound random nodes if repairs are scheduled automatically across the cluster.
• Gossip flapping: If nodes have intermittent network issues, gossip will retry connections constantly, overloading the InternalResponseStage.
• Bad client queries: Bursts of full-table scans, wide range queries, or requests with ALL consistency level force nodes to coordinate with multiple peers, spiking CPU. Timeout retries from clients make this worse.

5. Verify System-Level Resource Contention #

Don't overlook non-Cassandra factors:

• Disk IO bottlenecks: Slow disk IO can make Cassandra threads appear to hog CPU while waiting for data. Use iostat to check if disk await times exceed 10ms or util% hits 100%—this often cascades to CPU spikes.
• External process interference: Check if monitoring scripts, backup tools, or other system processes are running on the affected nodes during spikes. Use top or htop to confirm it's actually the Cassandra process eating CPU.
• Network saturation: If a node's network bandwidth is maxed out, data transfers stall, leaving Cassandra threads stuck waiting and consuming CPU. Use iftop to check for traffic spikes.

Since the spikes are random, prioritize looking for periodic tasks (auto-repairs, scheduled queries) or transient traffic bursts. Your existing logs and stats should help narrow down which of these areas is the culprit.

内容的提问来源于stack exchange，提问作者Evgeni