单节点Elasticsearch 7.17.0集群无法进入Green状态的解决问询

问题现象 #

• 集群持续处于Yellow状态，unassigned_shards值为1
• 未分配分片属于数据流的写入索引，直接删除索引报错（提示该索引是数据流的写入索引，无法删除）
• 分片分配解释显示：主分片已在当前节点，ES不允许将副本分片分配到同一节点，导致副本分片无法分配

配置信息 #

使用的ConfigMap配置如下：

apiVersion: v1
data:
  elasticsearch.yml: |-
    discovery:
      type: single-node
    network:
      host: 0.0.0.0
    path:
      data: /bitnami/elasticsearch/data
    xpack:
      ml:
        enabled: false
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/instance: elasticsearch
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: elasticsearch
    argocd.argoproj.io/instance: elasticsearch
    helm.sh/chart: elasticsearch-19.5.5
  name: elasticsearch
  namespace: elasticsearch

关键查询结果 #

1. 集群健康状态查询：

curl -XGET http://localhost:9200/_cluster/health?pretty=true
{
  "cluster_name" : "elasticsearch",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 7,
  "active_shards" : 7,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 1,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 87.5
}

2. 分片状态查询（未分配的是副本分片）：

curl -XGET http://localhost:9200/_cat/shards
magento2_product_1_v583                                       0 p STARTED    4868 18.9mb 10.110.4.229 elasticsearch-master-0
.ds-ilm-history-5-2022.10.28-000014                           0 p STARTED                10.110.4.229 elasticsearch-master-0
.ds-ilm-history-5-2022.11.27-000015                           0 p STARTED                10.110.4.229 elasticsearch-master-0
.ds-ilm-history-5-2022.08.29-000010                           0 p STARTED                10.110.4.229 elasticsearch-master-0
.ds-ilm-history-5-2022.09.28-000012                           0 p STARTED                10.110.4.229 elasticsearch-master-0
.geoip_databases                                              0 p STARTED      40 38.1mb 10.110.4.229 elasticsearch-master-0
.ds-.logs-deprecation.elasticsearch-default-2022.12.21-000022 0 p STARTED                10.110.4.229 elasticsearch-master-0
.ds-.logs-deprecation.elasticsearch-default-2022.12.21-000022 0 r UNASSIGNED 

3. 删除索引失败的返回结果：

curl -XGET http://localhost:9200/_cat/shards | grep UNASSIGNED | awk {'print $1'} | xargs -i curl -XDELETE "http://localhost:9200/{}"
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"index [.ds-.logs-deprecation.elasticsearch-default-2022.12.21-000022] is the write index for data stream [.logs-deprecation.elasticsearch-default] and cannot be deleted"}],"type":"illegal_argument_exception","reason":"index [.ds-.logs-deprecation.elasticsearch-default-2022.12.21-000022] is the write index for data stream [.logs-deprecation.elasticsearch-default] and cannot be deleted"},"status":400}

4. 分片分配解释结果：

curl -XGET http://localhost:9200/_cluster/allocation/explain?pretty=true | jq
{
  "note": "No shard was specified in the explain API request, so this response explains a randomly chosen unassigned shard. There may be other unassigned shards in this cluster which cannot be assigned for different reasons. It may not be possible to assign this shard until one of the other shards is assigned correctly. To explain the allocation of other shards (whether assigned or unassigned) you must specify the target shard in the request to this API.",
  "index": ".ds-.logs-deprecation.elasticsearch-default-2022.12.21-000022",
  "shard": 0,
  "primary": false,
  "current_state": "unassigned",
  "unassigned_info": {
    "reason": "CLUSTER_RECOVERED",
    "at": "2022-12-25T07:52:37.022Z",
    "last_allocation_status": "no_attempt"
  },
  "can_allocate": "no",
  "allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes",
  "node_allocation_decisions": [
    {
      "node_id": "aURccTcnSuqPC3fBfmezCg",
      "node_name": "elasticsearch-master-0",
      "transport_address": "10.110.4.229:9300",
      "node_attributes": {
        "xpack.installed": "true",
        "transform.node": "true"
      },
      "node_decision": "no",
      "deciders": [
        {
          "decider": "same_shard",
          "decision": "NO",
          "explanation": "a copy of this shard is already allocated to this node [[.ds-.logs-deprecation.elasticsearch-default-2022.12.21-000022][0], node[aURccTcnSuqPC3fBfmezCg], [P], s[STARTED], a[id=tsxhnODlSn-i__-vEvJj3A]]"
        }
      ]
    }
  ]
}

解决方案 #

问题核心是单节点环境下，部分索引的副本分片数仍为1，但没有额外节点可分配副本，导致分片无法分配。

执行以下命令将所有索引的副本分片数设置为0：

curl -v -XPUT "localhost:9200/*/_settings" -H 'Content-Type: application/json' -d '
{
  "index" : {
    "number_of_replicas" : 0
  }
}
'

执行后返回{"acknowledged":true}，再次查询索引状态，集群已恢复为Green：

curl -XGET http://localhost:9200/_cat/indices     
green open magento2_product_1_v583 hvYpUxJUT16-g6_YS8qkaA 1 0 4868  0 18.9mb 18.9mb
green open .geoip_databases        tDXBLQRdSFeQyi6Pk5zq2Q 1 0   40 40 38.1mb 38.1mb

内容的提问来源于stack exchange，提问作者DmitrySemenov