如何在OpenSearch查询中为指定match字段设置size选项

现有日志示例 #

以下是OpenSearch中logs-test索引的部分日志数据：

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 5,
      "relation": "eq"
    },
    "max_score": null,
    "hits": [
      {
        "_index": "logs-test",
        "_id": "NUWi8IYBR8llKo4TWxVe",
        "_score": null,
        "_source": {
          "@timestamp": "2023-03-17T17:33:20.000000000Z",
          "Body": {
            "active_connections": "415",
            "app_name": "app1",
            "attack_event": "Attack started"
          },
          "Name": "app1-logs"
        }
      },
      {
        "_index": "logs-test",
        "_id": "NUWi8IYBR8llKo4TWxVe",
        "_score": null,
        "_source": {
          "@timestamp": "2023-03-17T17:33:30.000000000Z",
          "Body": {
            "active_connections": "415",
            "app_name": "app1",
            "attack_event": "Under Attack"
          },
          "Name": "app1-logs"
        }
      },
      {
        "_index": "logs-test",
        "_id": "NUWi8IYBR8llKo4TWxVe",
        "_score": null,
        "_source": {
          "@timestamp": "2023-03-17T17:33:40.000000000Z",
          "Body": {
            "active_connections": "415",
            "app_name": "app1",
            "attack_event": "Under Attack"
          },
          "Name": "app1-logs"
        }
      },
      {
        "_index": "logs-test",
        "_id": "NUWi8IYBR8llKo4TWxVe",
        "_score": null,
        "_source": {
          "@timestamp": "2023-03-17T17:33:50.000000000Z",
          "Body": {
            "active_connections": "415",
            "app_name": "app1",
            "attack_event": "Under Attack"
          },
          "Name": "app1-logs"
        }
      },
      {
        "_index": "logs-test",
        "_id": "NUWi8IYBR8llKo4TWxVe",
        "_score": null,
        "_source": {
          "@timestamp": "2023-03-17T17:34:10.000000000Z",
          "Body": {
            "active_connections": "415",
            "app_name": "app1",
            "attack_event": "Attack ended"
          },
          "Name": "app1-logs"
        }
      }
    ]
  }
}

需求 #

• 保留所有attack_event为Attack started和Attack ended的记录
• 仅返回attack_event为Under Attack的最新1条记录（按@timestamp倒序取）
• 最终结果按@timestamp升序排列

当前问题 #

原查询仅能通过全局size参数限制返回总数，无法针对特定attack_event类型单独设置返回数量：

curl -X GET http://<OPENSEARCH_HOST:PORT>/logs-test/_search?pretty -H "Content-Type: application/json" -d '{"query": {"bool": {"should": [{"match": {"Body.attack_event": "Attack started"}}, {"match": {"Body.attack_event": "Attack ended"}}, {"match": {"Body.attack_event": "Under Attack"}}]}}, "sort": [{"@timestamp": {"order": "asc"}}], "size": 2}'

解决方案 #

方案1：多搜索（Multi-Search） #

通过一次请求发起三个独立查询，分别获取不同事件类型的结果，再在客户端合并排序，最贴合需求：

curl -X GET http://<OPENSEARCH_HOST:PORT>/_msearch?pretty -H "Content-Type: application/json" -d '
{"index": "logs-test"}
{"query": {"term": {"Body.attack_event.keyword": "Attack started"}}, "sort": [{"@timestamp": "asc"}], "size": 100}
{"index": "logs-test"}
{"query": {"term": {"Body.attack_event.keyword": "Under Attack"}}, "sort": [{"@timestamp": "desc"}], "size": 1}
{"index": "logs-test"}
{"query": {"term": {"Body.attack_event.keyword": "Attack ended"}}, "sort": [{"@timestamp": "asc"}], "size": 100}
'

结果处理

将返回的三个结果集的hits数组合并，再按@timestamp升序排序，即可得到期望的结构。

方案2：聚合+Top Hits #

通过terms聚合按事件类型分组，为每个组配置top_hits聚合精准控制返回数量，适合需要结构化分组结果的场景：

curl -X GET http://<OPENSEARCH_HOST:PORT>/logs-test/_search?pretty -H "Content-Type: application/json" -d '{
  "size": 0,
  "query": {
    "bool": {
      "filter": {
        "terms": {"Body.attack_event.keyword": ["Attack started", "Attack ended", "Under Attack"]}
      }
    }
  },
  "aggs": {
    "event_groups": {
      "terms": {
        "field": "Body.attack_event.keyword",
        "include": ["Attack started", "Attack ended", "Under Attack"]
      },
      "aggs": {
        "filtered_hits": {
          "top_hits": {
            "size": "{{#eq _key 'Under Attack'}}1{{else}}100{{/eq}}",
            "sort": [{"@timestamp": {"order": "{{#eq _key 'Under Attack'}}desc{{else}}asc{{/eq}}"}}]
          }
        }
      }
    }
  }
}'

结果处理

遍历聚合结果中的每个bucket，提取filtered_hits.hits数据合并即可。

内容的提问来源于stack exchange，提问作者Srikanth Pulletikurthi