Elasticsearch的enrich.fetch_size参数能否调大？调大报错求助

enrich.fetch_size是将源索引重索引到enrich索引时的最大批量大小，默认值为10000。当在elasticsearch.yml中将其值改为20000后，执行ingest policy时触发如下错误：

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "Batch size is too large, size must be less than or equal to: [10000] but was [20000]. Scroll batch sizes cost as much memory as result windows so they are controlled by the [index.max_result_window] index level setting."
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "Partial shards failure",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "name-of-index",
        "node" : "node-id",
        "reason" : {
          "type" : "illegal_argument_exception",
          "reason" : "Batch size is too large, size must be less than or equal to: [10000] but was [20000]. Scroll batch sizes cost as much memory as result windows so they are controlled by the [index.max_result_window] index level setting."
        }
      }
    ]
  },
  "status" : 400
}

你的配置文件片段：

...
discovery:
  seed_hosts:
    - "127.0.0.1"
    - "[::1]"
    - elasticsearch

script:
  context:
    template:
      max_compilations_rate: 400/5m
      cache_max_size: 400

enrich:
  fetch_size: 20000
...

解决方案 #

可以调大enrich.fetch_size，但需要先调整源索引的index.max_result_window参数，使其值不小于你设置的enrich.fetch_size（这里就是20000）。因为滚动批量的内存消耗和结果窗口一致，所以该参数受索引级别的index.max_result_window限制，默认值为10000。

执行以下命令修改源索引的设置：

PUT /name-of-index/_settings
{
  "index.max_result_window": 20000
}

注意事项 #

• 如果有多个源索引需要用于enrich policy，每个索引都要单独调整index.max_result_window值。
• 增大该值会增加节点的内存占用，需要根据集群的实际内存资源合理设置，避免引发内存不足问题。

内容的提问来源于stack exchange，提问作者kingkong