如何从Elasticsearch索引获取唯一文档？是否需全字段聚合查询？

Hey there! Let's tackle your questions step by step—this is a common scenario, and you definitely don't have to jump through hoops with per-field aggregations.

Do I need to write aggregations for all fields to get unique documents? #

Absolutely not. Aggregating every field would be wildly inefficient, especially on large indices. It would tax your cluster's resources unnecessarily and result in a slow, unwieldy query. You only need to target the field(s) that define a "unique" document for your use case (e.g., a business ID, a combination of user ID + timestamp, etc.).

How can I get unique documents without aggregating every field? #

There are two straightforward, efficient approaches depending on your data structure:

1. Use collapse for a single unique identifier #

If you have a single field that uniquely identifies your documents (like a user_id, order_id, or a custom business key), the collapse feature is your best bet. It groups documents by the specified field and returns one document per group.

Here's an example query:

{
  "query": {
    "match_all": {}
  },
  "collapse": {
    "field": "business_id.keyword" // Use .keyword for text fields to avoid tokenization
  },
  "size": 10000, // Adjust based on how many unique docs you need
  "sort": [{"created_at": "desc"}] // Optional: Control which document from each group is returned
}

If you need to fetch more unique documents than the size limit, pair this with a scroll or point-in-time (PIT) to paginate through results.

2. Combine fields into a unique key with script-based terms aggregation #

If uniqueness is defined by a combination of fields (e.g., user_id + product_id), use a script to concatenate those fields into a single unique key, then aggregate on that key. Add a top_hits sub-aggregation to retrieve the full document for each unique group.

Example query:

{
  "size": 0, // We don't need the raw hits, just the aggregation results
  "aggs": {
    "unique_documents": {
      "terms": {
        "script": "doc['user_id.keyword'].value + '|' + doc['product_id.keyword'].value",
        "size": 10000 // Adjust based on expected unique count
      },
      "aggs": {
        "top_document": {
          "top_hits": {
            "size": 1,
            "_source": ["user_id", "product_id", "details"] // Optional: Specify fields to return
          }
        }
      }
    }
  }
}

This approach avoids nested aggregations on every field and focuses only on the combination that defines uniqueness.

Key Takeaway #

You never need to aggregate every field to get unique documents. Focus on the field(s) that logically define uniqueness for your use case, and use either collapse (for single fields) or a scripted terms aggregation (for field combinations) to efficiently retrieve the unique documents you need.

内容的提问来源于stack exchange，提问作者Akib Ali