Cassandra节点增减是否需全集群节点在线？迁移问题咨询

Great questions—let's break these down one by one, since Cassandra's cluster operations have some nuanced behavior around node availability.

Short answer: No, you don't need every single node online, but you do need enough nodes available to meet the cluster's consistency requirements (specifically, a quorum of nodes for the system keyspace).

• When adding a new node: The cluster uses gossip to spread the new node's token range and coordinate data migration. As long as a majority of nodes are online (so the cluster can reach consensus on the topology change), the new node can join successfully. Offline nodes will catch up on the topology change once they come back online via gossip.
• When removing a node:
  • If decommissioning a live node: It will stream its data to other online nodes. You don't need all nodes online, just enough healthy nodes to receive the data (matching your replication factor). The decommission will stall if there aren't enough online peers to accept the data.
  • If removing an offline node: Use nodetool removenode to force its removal, but you still need a quorum of nodes online to agree to the topology change.

Adding a node with offline peers #

When you add a new node, existing nodes only stream data to it if they're online at the time of the add. If some nodes were offline, their portion of the data that falls into the new node's token range won't be migrated automatically when those nodes come back online.

Here's how to fix this:

• Once the offline nodes are back up and synced with the cluster, run nodetool rebuild on the new node. This command tells the new node to pull all data it should own from available peers (including the recently recovered ones). You can optionally target a specific recovered node with nodetool rebuild --source <node_ip>, but leaving the source blank will let it pull from all healthy nodes.
• A full nodetool repair on the new node also works, but rebuild is more efficient because it only syncs the data the new node is responsible for, rather than validating all cluster data.

Will it happen automatically? No, Cassandra doesn't trigger this sync on its own when offline nodes recover. You need to initiate the rebuild or repair manually to fill in the missing data.

Removing a node with offline peers #

If you're decommissioning a live node and some target nodes (that should receive the decommissioned node's data) are offline, the decommission process will stall or fail—Cassandra won't complete it until it can stream all data to online nodes.

If you use nodetool removenode to force the removal of a node while peers are offline, you risk data loss: any data that was supposed to be replicated to the offline nodes won't have a valid home after the node is removed. If those offline nodes never come back, that data is permanently lost.

To avoid or fix this:

• Always try to decommission nodes when most peers are online, so all data can be streamed to healthy nodes without gaps.
• If you had to force-remove a node while peers were offline, once those offline nodes recover, run nodetool repair on the affected keyspaces. This will sync the missing data across the cluster and restore the lost replicas—assuming the offline nodes still have the intact data.

内容的提问来源于stack exchange，提问作者kingluo