Logstash Elasticsearch Filter查询模板语法错误，求正确Match写法

问题场景 #

使用Logstash的Elasticsearch filter，通过query_template（sample.json）的query_string查询获取主机名时出现语法解析错误，但在Kibana Dev Tools中使用match查询可正常运行。

原错误的query_template（sample.json） #

{
  "size": 1,
  "query": {
    "query_string": {
      "query": "[local_metadata.host.id]:%{[host.id]}"
    }
  }
}

错误警告信息 #

[WARN ] 2023-04-28 09:59:01.047 [[main]>worker1] elasticsearch - Failed to query elasticsearch for previous event {:index=>".fleet-agents", :error=>"[400] {\"error\":{\"root_cause\":[{\"type\":\"query_shard_exception\",\"reason\":\"Failed to parse query [[local_metadata.host.id]:%{[host.id]}]\",\"index_uuid\":\"3_qR6WxhRRugbHtormpQkg\",\"index\":\".fleet-agents-7\"}],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[{\"shard\":0,\"index\":\".fleet-agents-7\",\"node\":\"ECbgB995T6OYF-0I-sus6A\",\"reason\":{\"type\":\"query_shard_exception\",\"reason\":\"Failed to parse query [[local_metadata.host.id]:%{[host.id]}]\",\"index_uuid\":\"3_qR6WxhRRugbHtormpQkg\",\"index\":\".fleet-agents-7\",\"caused_by\":{\"type\":\"parse_exception\",\"reason\":\"Cannot parse '[local_metadata.host.id]:%{[host.id]}': Encountered \\\" \\\" ]\\\" \\\" ] \\\"\\\" at line 1, column 23.\nWas expecting:\n    \\\"TO\\\" ...\n    \"},\"caused_by\":{\"type\":\"parse_exception\",\"reason\":\"Encountered \\\" \\\" ]\\\" \\\" ] \\\"\\\" at line 1, column 23.\nWas expecting:\n    \\\"TO\\\" ...\n    \"}}}}]},\"status\":400}"

Dev Tools中可正常运行的Match查询示例 #

GET .fleet-agents/_search
{
 "query": {
   "match": {
     "local_metadata.host.id": "222222-222-2222-22-222"
   }
 } 
}

Logstash Pipeline完整配置 #

input {
        elasticsearch
        {
        hosts => "localhost"
        user => "reader"
        password => "**************"
        index => "*-test"
        query => '{ "query": {
                             "bool": {
                                      "must": [{"terms": { "kibana.alert.severity": [ "high", "critical"] }} ],
                                      "filter": [ {"range": {"@timestamp": { "gte": "now-2w"}}}] 
                                     }
                             }
                  }'
        schedule => "/1 * * * *"
        size => 500
        scroll => "5m"
        docinfo => true
        docinfo_target => "[@metadata][doc]"
        codec => "json"
        }
    }


filter {
elasticsearch {
              hosts => "localhost"
              user => "fleet-user"
              password => "************"
              index => ".fleet-agents"
              query_template => "/data/logstash/pipelines/sample.json"
              fields => { "host.name" => "host_name" }
              }

mutate {
        add_field => {
            "alertHostName" => "%{[host_name]}"
            "alertReason" => "%{kibana.alert.reason}"
            "alertSeverity" => "%{kibana.alert.severity}"
            "alertTime" => "%{kibana.alert.original_time}"

        }
    }

}

output {
stdout {
 codec => "json"
}
}

解决方案：正确的Match查询模板 #

将sample.json修改为结构化的match查询，避免query_string的语法解析问题：

{
  "size": 1,
  "query": {
    "match": {
      "local_metadata.host.id": "%{[host.id]}"
    }
  }
}

说明 #

原query_string报错是因为Elasticsearch解析字符串查询时，对带点的字段名和Logstash变量占位符的组合解析失败，换成结构化的match查询后，直接指定字段和变量，无需字符串语法解析，即可正常匹配local_metadata.host.id字段并替换Logstash的%{[host.id]}变量。

内容的提问来源于stack exchange，提问作者NiveditaK