Let’s walk through how to figure out why your cluster has uneven load (with nodes hitting 1.39 TB and 1.03 TB while others sit at 700-900 GB) and get everything balanced again.

Step 1: Diagnose the Root Cause #

Check Effective Node Ownership #

Your nodetool status output shows Owns (effective) as ?—that means we don’t have the actual ownership percentages needed to spot token skew. Run this command to get accurate data:

nodetool status -r

With 10 nodes and 32 vnodes each, every node should own roughly 10% of the cluster’s data. If the overloaded nodes have way higher percentages (15%+), uneven token distribution is the issue.

Hunt for Oversized Partitions #

The #1 cause of data skew in Cassandra is poorly designed partition keys creating massive individual partitions. To find the culprits:

• Use nodetool toppartitions to list the largest partitions by size:

  nodetool toppartitions -k YOUR_KEYSPACE -t YOUR_TABLE -s size -n 10
  

• Or use nodetool cfstats for a specific table to compare average vs. max partition size:

  nodetool cfstats YOUR_KEYSPACE.YOUR_TABLE
  

If you see a partition that’s 10x+ larger than the average, that’s your problem—this single partition is weighing down the nodes that host it.

Rule Out Ongoing Cluster Changes #

If you’ve added or removed nodes recently, Cassandra’s automatic balancing might still be in progress. Check for active data streaming with:

nodetool netstats

Or get a detailed balance progress report:

nodetool balanceinfo

Step 2: Fix the Skew and Balance the Cluster #

Case 1: Uneven Token Distribution #

If nodetool status -r shows skewed ownership percentages:

• Trigger a manual rebalance to redistribute vnodes evenly across the cluster:

  nodetool rebalance
  

  Note: This will start streaming data between nodes, so run it during low-traffic hours to avoid performance hits.
• Verify no manual token assignments were made (vnodes handle token distribution automatically by default). Ensure all nodes have the same num_tokens setting (32, as shown in your status) in cassandra.yaml.

Case 2: Large/Hot Partitions (Most Likely Scenario) #

If you found oversized partitions, you’ll need to fix your data model to split them up:

1. Redesign the Partition Key: Add a sharding component to break large partitions into smaller, manageable ones. For example:
   • If your original key was user_id (and one user has 1TB of data), switch to a composite key like (user_id, shard_id) where shard_id is a hash of the user ID (e.g., user_id % 10 to split into 10 partitions).
   • If time-based data is skewing (e.g., one day’s data is massive), split by hour or minute instead of day.
2. Migrate Data: Create a new table with the improved schema, then move data over using tools like Spark, sstableloader, or a custom CQL script.
3. Clean Up: Once data is migrated, drop the old table and run nodetool cleanup on all nodes to remove stale data from token ranges they no longer own.

General Best Practices for Long-Term Balance #

• Keep automatic compaction enabled (default setting: enable_auto_compaction: true in cassandra.yaml) to maintain efficient disk usage.
• Run nodetool cleanup regularly after cluster changes (add/remove nodes) to free up unused disk space.
• Set up alerts for when a node’s load is 20%+ above the cluster average to catch skew early.

内容的提问来源于stack exchange，提问作者Firdousi Farozan