MongoDB分片集群存储分配及新增分片数据迁移问题咨询

Hey there, let's tackle your questions step by step based on your current cluster setup (2 mongos routers, 2 replicated config servers, 6 un-replicated shards):

1. Will new shards automatically get data from existing shards? #

Yes, but with key caveats tied to your specific setup. When you add a new shard to your cluster, MongoDB's built-in Balancer will detect the imbalance in data distribution across shards. It will automatically start migrating chunks (the basic unit of data in sharding) from overloaded existing shards to the new empty shard.

However, as you noted, this process can be slow for two critical reasons:

• The balancer only targets chunks from the sharded collections you've configured (like blockchains.ethereum_balance and blockchains.ethereum_daily), so it won't migrate data from unsharded collections.
• If the target shard doesn't have the required indexes for the sharded collection, MongoDB will first create those indexes on the new shard before starting any chunk migration. Index creation on large datasets adds significant overhead, which is a major contributor to your slow migration times.

2. How to manage storage distribution effectively? #

Given your current challenges (slow auto-migration, missing indexes on some shards, uneven storage), here are actionable steps:

- Fix index consistency first #

Since indexes are only present on a single shard, you need to ensure all shards have the required indexes for your sharded collections. Run these commands on a mongos instance to sync indexes across all shards:

// Sync index for ethereum_balance collection
db.blockchains.ethereum_balance.createIndexes([{"dapp_id": "hashed"}])

// Sync index for ethereum_daily collection
db.blockchains.ethereum_daily.createIndexes([{"to": "hashed"}])

Running these on mongos will propagate index creation to all shards, which is a prerequisite for smooth chunk migration.

- Optimize balancer behavior #

• Verify balancer status: First confirm the balancer is running with sh.getBalancerState(). If it's disabled, enable it with sh.startBalancer().
• Adjust chunk size (if needed): The default chunk size is 64MB. For large datasets, smaller chunks can speed up migration (avoid making them too small, as it increases cluster overhead). Adjust it with:

  sh.setBalancerChunkSizeMB(32) // Example: set to 32MB
  

• Schedule migration during off-peak hours: If auto-migration impacts your application, set a balancer window to limit migration to low-traffic times:

  sh.setBalancerWindow({ start: "22:00", stop: "06:00" }) // Uses UTC time
  

- Manual chunk migration (for targeted balancing) #

If the auto-balancer is too slow or not prioritizing the right chunks, you can manually relocate specific chunks. First, use sh.status() to identify overloaded shards and target chunks, then use:

// Example: Move a chunk of ethereum_balance to the new shard
sh.moveChunk("blockchains.ethereum_balance", {dapp_id: "hashed_value_of_target_key"}, "new_shard_name")

Note: Use manual migration sparingly, as it requires careful monitoring to avoid disrupting cluster stability.

- Monitor distribution regularly #

Use sh.status() to check chunk distribution across shards, and db.collection.getShardDistribution() to view storage usage per shard for specific collections. This helps you catch imbalances early and adjust your strategy accordingly.

内容的提问来源于stack exchange，提问作者pr0logas