不同配置机器搭建Elasticsearch集群的影响及滚动升级规划咨询

Great question! Let's walk through the key impacts of running a cluster with mismatched data nodes (Node A and Node B) plus your specific setup with a low-resource master-only candidate (Node M), along with some actionable tips for your use case.

Key Impacts to Watch For #

1. Uneven Shard Load & Performance Bottlenecks #

Elasticsearch tries to distribute shards evenly across data nodes by default, but this doesn't account for hardware differences. Your high-spec Node B will handle far more shard load (queries, indexing, heap usage) than the low-spec Node A without breaking a sweat. Meanwhile, Node A will quickly become the cluster's performance bottleneck:

• It may struggle with CPU/memory limits during peak traffic, leading to slow query responses or indexing delays.
• Frequent garbage collection (GC) pauses are likely on Node A due to limited heap, which can disrupt cluster stability and cause timeouts.

2. Heap Memory Disparities Cause Cache Inefficiencies #

Heap memory is critical for Elasticsearch's in-memory caches (field data cache, query cache, request cache). Node A's smaller heap will:

• Have less space for these caches, leading to lower cache hit rates. This forces the node to re-compute data more often, worsening performance.
• Be at higher risk of out-of-memory (OOM) errors if shard memory usage spikes (e.g., during large aggregations or bulk indexing).

3. Master Node Stability Risks #

Your Node M is even lower-spec than Node A, and while it doesn't store data, master nodes handle cluster metadata management, shard allocation decisions, and election coordination. If Node M gets elected as the master:

• Cluster administrative operations (like updating mappings, reallocating shards) will slow down significantly.
• Under high cluster load, Node M might struggle to keep up with state updates, leading to cluster instability or even unexpected master elections.
• Even if Node A is elected master, it has to balance master duties with data node workloads—its low specs make this a risky combination that can throttle cluster responsiveness.

4. Rolling Upgrade Challenges #

You mentioned supporting rolling upgrades, but mismatched nodes complicate this process:

• When you restart Node B for an upgrade, all its shards will fail over to Node A. Node A's limited resources may not handle the sudden extra load, causing performance drops or even shard unavailability.
• If the master node switches to Node M during the upgrade, the cluster's ability to manage the shard failover and upgrade process will be impaired.

Actionable Recommendations for Your Setup #

• Tune Shard Allocation to Match Node Capacity:
  Use node attributes and allocation weighting to direct more shards to Node B. For example:

  1. Add attributes to your node configs:

     # Node B config
     node.attr.hardware: high
     # Node A config
     node.attr.hardware: low
     

  2. Configure allocation weighting to prioritize Node B:

     cluster.routing.allocation.node.weight.high: 3
     cluster.routing.allocation.node.weight.low: 1
     

  This tells Elasticsearch to assign 3x more shards to Node B than Node A, balancing load based on hardware.

• Reassess Master Node Candidates:
  Node M's low specs make it a poor master candidate. Instead:

  • Set Node M to a voting-only node (if using Elasticsearch 7.0+):

    node.master: true
    node.data: false
    node.voting_only: true
    

  This lets it participate in elections without ever becoming the active master. If you're on an older version, remove Node M's node.master: true setting entirely—stick with Node A and Node B as master candidates, since they have better resources to handle master duties.

• Optimize Node A's Resources:

  • Limit Node A's heap to a safe size (max 50% of its physical memory, never over 32GB).
  • Disable non-critical caches if possible (e.g., indices.queries.cache.enabled: false if your workload doesn't rely heavily on repeated queries).
  • Use index-level settings to reduce memory pressure (e.g., set index.codec: best_compression to lower disk/heap usage for infrequently accessed indices).

• Test Rolling Upgrades in Staging:
  Before upgrading production, simulate the process:

  • Temporarily disable shard allocation, restart Node B, and monitor Node A's CPU/memory/GC metrics. If Node A can't handle the failover load, consider temporarily scaling up Node A or adjusting shard counts before the production upgrade.

内容的提问来源于stack exchange，提问作者seekahead