如何在Elasticsearch JS（AWS环境）中按标题字段去重查询结果

Hey Zach, let's tackle this duplicate title issue you're facing with Elasticsearch JS in AWS Lambda!

The older aggregation-based approaches you found are indeed outdated for modern Elasticsearch versions, and they're not ideal here anyway—since you want to return actual deduplicated documents instead of just statistical counts. The right tool for this job is Elasticsearch's collapse parameter, which was designed specifically to group results by a field and return only the top document from each group.

Here's how to modify your query to deduplicate by title #

First, make sure you're targeting the keyword version of your data.title field (if it exists). Text fields are analyzed, so using them for deduplication can lead to unexpected results (e.g., "My Title" and "my title" might be treated as different). If your data.title field is mapped as a text type, it likely has a keyword subfield (common default mapping) that we can use for exact-match deduplication.

Update your search query like this:

const { Client } = require('@elastic/elasticsearch');
// Initialize your ES client with AWS config (credentials, node URL, etc.)
const client = new Client({
  node: 'your-aws-es-endpoint',
  auth: { /* your AWS auth setup */ }
});

async function runSearch(data) {
  const result = await client.search({
    index: 'your-target-index', // Replace with your index name
    query: {
      bool: {
        must: [{
          multi_match: {
            query: data.searchTerm,
            fields: ["data.address", "data.category", "data.description^1.0", "data.order_id", "data.owner", "data.sub_title^1.5", "data.tags*^1.5", "data.title^2.5", "data.token_id", "data.txhash", "data.url", "data.store.title"],
            operator: "or",
            lenient: "true",
            type: 'best_fields',
            fuzziness: "AUTO"
          }
        }]
      }
    },
    from: Number(data.lastKey),
    size: Number(data.size), // This will now return 25 deduplicated results
    collapse: {
      field: 'data.title.keyword' // Use keyword field for exact deduplication
      // Optional: Add inner_hits if you want to see all duplicate docs for a title
      // inner_hits: { size: 5 }
    }
  });

  return result.hits.hits;
}

Why this works #

• The collapse parameter groups all matching documents by the specified data.title.keyword field.
• For each group, it only returns the document with the highest relevance score (matching your original multi-match query logic).
• Unlike aggregations, this directly returns the deduplicated document data you need, rather than just counts or buckets.

Quick check if you don't have a keyword subfield #

If your data.title field doesn't have a keyword subfield, you'll need to update your index mapping to add it (you can do this without reindexing existing data, though new docs will use it immediately):

PUT /your-target-index/_mapping
{
  "properties": {
    "data": {
      "properties": {
        "title": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256 // Adjust based on your title length
            }
          }
        }
      }
    }
  }
}

If you can't modify the mapping right now, you can try using data.title directly in the collapse field, but be aware that analyzed text may lead to inconsistent deduplication.

内容的提问来源于stack exchange，提问作者Zach