Elasticsearch 7.9.0集群排除节点范围时首尾节点分片未迁移问题

问题现象 #

在K8s环境下的64节点Elasticsearch集群中，尝试分批替换所有存储。当执行节点排除配置（比如排除前32个节点）时，集群会将排除列表中大部分节点的分片迁移到可用节点，但列表首尾的节点（如node1和node32）上的分片始终不会被迁移。无论设置哪个节点范围，首尾节点的分片都保持原地不动。

执行的节点排除命令 #

PUT _cluster/settings
{
 "persistent" : {
   "cluster.routing.allocation.exclude._name" : [ "node1" ,"node2" ... ,"node32"]
 }
}

尝试排查时的错误响应 #

错误1：请求路径错误 #

执行GET cluster/allocation/explain时返回404：

#! Deprecation: [types removal] Specifying types in document get requests is deprecated, use the /{index}/_doc/{id} endpoint instead.
{
 "error" : {
 "root_cause" : [
  {
    "type" : "index_not_found_exception",
    "reason" : "no such index [cluster]",
    "resource.type" : "index_expression",
    "resource.id" : "cluster",
    "index_uuid" : "_na_",
    "index" : "cluster"
  }
],
"type" : "index_not_found_exception",
"reason" : "no such index [cluster]",
"resource.type" : "index_expression",
"resource.id" : "cluster",
"index_uuid" : "_na_",
"index" : "cluster"
},
"status" : 404
}

说明：此错误是因为请求路径写错，正确路径应为GET _cluster/allocation/explain（开头的下划线不能省略）。

错误2：无未分配分片可解释 #

执行正确路径GET _cluster/allocation/explain时返回400：

{
"error" : {
"root_cause" : [
  {
    "type" : "illegal_argument_exception",
    "reason" : "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"
  }
],
"type" : "illegal_argument_exception",
"reason" : "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"
},
 "status" : 400
}

错误3：需指定具体分片 #

添加参数include_yes_decisions=true后仍返回400：

{
"error" : {
"root_cause" : [
  {
    "type" : "illegal_argument_exception",
    "reason" : "No shard was specified in the request which means the response should explain a randomly-chosen unassigned shard, but there are no unassigned shards in this cluster. To explain the allocation of an assigned shard you must specify the target shard in the request."
  }
],
"type" : "illegal_argument_exception",
"reason" : "No shard was specified in the request which means the response should explain a randomly-chosen unassigned shard, but there are no unassigned shards in this cluster. To explain the allocation of an assigned shard you must specify the target shard in the request."
},
"status" : 400
}

可能原因及解决方法 #

• 分片分配规则限制：首尾节点上的分片可能因为副本数限制、索引级别的分配过滤规则（如index.routing.allocation.*配置）导致无法迁移。比如某分片是唯一主分片，且集群中无其他节点满足其分配条件（节点标签匹配、磁盘空间足够等），Elasticsearch会避免强制迁移以防数据不可用。
• 节点名称匹配问题：检查排除列表中的节点名称是否与集群实际节点名称完全一致，注意大小写、特殊字符或K8s节点的后缀（如node1-es-0）是否导致匹配失败。
• 版本已知bug：Elasticsearch 7.9.0存在部分分片分配相关bug，批量排除节点时首尾节点的匹配逻辑可能异常，建议升级到该大版本的最新补丁（如7.9.3）。
• 强制排查分片状态：针对首尾节点上的具体分片，使用指定分片的分配解释命令获取详细决策原因：

  GET _cluster/allocation/explain
  {
    "index": "your-index-name",
    "shard": 0,
    "primary": true
  }
  

  替换参数为目标分片信息，根据返回结果针对性解决。
• 临时强制迁移：确认分片可安全迁移后，使用重路由命令强制迁移：

  POST _cluster/reroute
  {
    "commands": [
      {
        "move": {
          "index": "your-index-name",
          "shard": 0,
          "from_node": "node1",
          "to_node": "node33"
        }
      }
    ]
  }
  

  注意：执行前需确保目标节点资源充足，避免集群负载过高。

内容的提问来源于stack exchange，提问作者R2D2