如何编写Elasticsearch多聚合查询获取各API的HTTP状态码计数并实现Kibana告警展示Top5热门API

Hey there! I totally get where you're coming from as an Elasticsearch beginner—nested aggregations can feel tricky at first, but they're exactly what we need here. Let's break down how to modify your existing query to get the per-API status code counts you're looking for.

Modified Elasticsearch Query #

We'll use a nested terms aggregation: first group all requests by the data.url field, then within each URL group, aggregate by data.response.status. Here's the full query:

{
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "@timestamp": {
              "from": "now-15m",
              "to": "now",
              "include_lower": true,
              "include_upper": true,
              "boost": 1
            }
          }
        }
      ],
      "adjust_pure_negative": true,
      "boost": 1
    }
  },
  "aggregations": {
    "API_URLs": {
      "terms": {
        "field": "data.url",
        "size": 10, // Adjust this to show more/less APIs as needed
        "min_doc_count": 1,
        "order": [
          { "_count": "desc" } // Sort APIs by total request volume (highest first)
        ]
      },
      "aggregations": {
        "Status_Codes": {
          "terms": {
            "field": "data.response.status",
            "size": 10,
            "min_doc_count": 1,
            "order": [
              { "_count": "desc" }
            ]
          }
        }
      }
    }
  }
}

Key Changes Explained: #

• Outer Aggregation (API_URLs): Groups all incoming requests by the data.url field, sorted by total request count so you can easily spot high-traffic APIs.
• Inner Aggregation (Status_Codes): Runs inside each URL group, counting how many times each HTTP status code appears for that specific API.

Expected Response Format #

The response will match the structure you requested, with each API URL followed by its status code breakdown:

"aggregations": {
  "API_URLs": {
    "doc_count_error_upper_bound": 0,
    "sum_other_doc_count": 12,
    "buckets": [
      {
        "key": "/search/results",
        "doc_count": 41,
        "Status_Codes": {
          "buckets": [
            { "key": 200, "doc_count": 30 },
            { "key": 201, "doc_count": 10 },
            { "key": 500, "doc_count": 1 }
          ]
        }
      },
      {
        "key": "/eligibility",
        "doc_count": 23,
        "Status_Codes": {
          "buckets": [
            { "key": 200, "doc_count": 20 },
            { "key": 500, "doc_count": 3 }
          ]
        }
      }
    ]
  }
}

Getting Top 5 APIs for Kibana Alerts #

To directly fetch the top 5 highest-traffic APIs, just set the size parameter in the API_URLs aggregation to 5. This will return only the URLs with the most requests, which you can use to build your Kibana alerts.

Quick Kibana Alert Setup Tip: #

1. In Kibana, create a Saved Search using the query above (with size:5 for the API aggregation).
2. Go to Stack Management > Alerts and Insights > Rules and create a new rule.
3. Use the saved search as your data source, then set conditions (e.g., "if 5xx status code count for /search/results exceeds 5 in 15 minutes").
4. Configure your preferred alert actions (Slack, email, etc.) and save the rule.

内容的提问来源于stack exchange，提问作者augustine vijay