如何通过Elasticsearch查询对比两个索引的文档字段差异？

完全可以通过Elasticsearch自身的查询/聚合能力实现，无需暴力遍历对比，以下是几种高效方案：

方案一：批量拉取+脚本字段标记差异（适合小批量文档） #

先批量获取所有unique_id，再通过_mget同时拉取两个索引的对应文档，用脚本字段直接判断是否存在差异：

1. 获取所有unique_id（已知所有id可跳过） #

GET /index_old/_search
{
  "size": 10000,
  "_source": ["unique_id"]
}

2. 批量对比文档 #

构造_mget请求，通过script_fields生成差异标识：

GET /_mget
{
  "docs": [
    {
      "_index": "index_old",
      "_id": "japanese_cheesecake",
      "_source": true,
      "script_fields": {
        "has_difference": {
          "script": {
            "source": "def old_desc = doc['description'].value; def old_ing = doc['ingredients'].value; def new_desc = params.new_doc.description; def new_ing = params.new_doc.ingredients; boolean diff = false; if (old_desc != new_desc) diff = true; if (!old_ing.equals(new_ing)) diff = true; return diff;",
            "params": {
              "new_doc": {
                "description": "Japanese cheesecake also known as soufflé-style cheesecake, cotton cheesecake, or light cheesecake is a variety of cheesecake that is usually lighter in texture and less sweet than North American-style cheesecakes",
                "ingredients": ["cream Cheese", "butter", "sugar", "egg", "butter"]
              }
            }
          }
        }
      }
    },
    // 其他unique_id的文档按此格式添加
    {
      "_index": "index_old",
      "_id": "turkish_delight",
      "_source": true,
      "script_fields": {
        "has_difference": {
          "script": {
            "source": "def old_desc = doc['description'].value; def old_ing = doc['ingredients'].value; def new_desc = params.new_doc.description; def new_ing = params.new_doc.ingredients; boolean diff = false; if (old_desc != new_desc) diff = true; if (!old_ing.equals(new_ing)) diff = true; if (params.new_doc.containsKey('origin') && !doc.containsKey('origin')) diff = true; return diff;",
            "params": {
              "new_doc": {
                "description": "Turkish delight or lokum is a family of confections based on a gel of starch and sugar",
                "ingredients": ["starch", "sugar", "pistachios", "dry fruits"],
                "origin": "turkey"
              }
            }
          }
        }
      }
    }
  ]
}

返回结果中has_difference为true的文档即为存在差异的，可直接提取对应内容分析具体差异。

方案二：跨索引聚合对比（适合大规模数据） #

利用Elasticsearch的聚合功能，按unique_id分组拉取两个索引的文档，通过脚本过滤出有差异的分组：

GET /index_old,index_new/_search
{
  "size": 0,
  "aggs": {
    "group_by_id": {
      "terms": {
        "field": "unique_id.keyword",
        "size": 10000
      },
      "aggs": {
        "get_docs": {
          "top_hits": {
            "size": 2,
            "_source": true
          }
        },
        "check_diff": {
          "bucket_script": {
            "buckets_path": {
              "old_doc": "get_docs.hits.hits[0]._source",
              "new_doc": "get_docs.hits.hits[1]._source"
            },
            "script": "def old = params.old_doc; def new = params.new_doc; boolean same = true; // 对比描述和食材 if (old.description != new.description) same = false; if (!old.ingredients.equals(new.ingredients)) same = false; // 检查新增字段 if (new.containsKey('origin') && !old.containsKey('origin')) same = false; return !same;"
          }
        }
      }
    },
    "filter_diff_groups": {
      "bucket_selector": {
        "buckets_path": {
          "is_diff": "check_diff"
        },
        "script": "params.is_diff == true"
      }
    }
  }
}

这个查询会直接返回所有存在差异的unique_id分组，以及对应的新旧文档内容，无需后续二次对比。

方案三：批量标记差异到新索引（需持久化结果时用） #

如果需要将差异结果保存到索引中，可使用_update_by_query结合脚本，从index_old获取对应文档并标记差异：

POST /index_new/_update_by_query
{
  "script": {
    "source": "def old_doc = ctx._source; def lookup_doc = params.lookup.get(index_old, ctx._id); ctx._source.has_diff = false; ctx._source.diff_details = []; // 对比字段 if (old_doc.description != lookup_doc.description) {ctx._source.has_diff = true; ctx._source.diff_details.add('description updated');} if (!old_doc.ingredients.equals(lookup_doc.ingredients)) {ctx._source.has_diff = true; ctx._source.diff_details.add('ingredients updated');} // 检查新增字段 if (old_doc.containsKey('origin') && !lookup_doc.containsKey('origin')) {ctx._source.has_diff = true; ctx._source.diff_details.add('origin field added');}",
    "params": {
      "lookup": {
        "index": "index_old",
        "type": "_doc"
      }
    }
  },
  "query": {
    "match_all": {}
  }
}

注：此方案需确保Elasticsearch开启了脚本的lookup功能（7.x+版本默认支持），执行后index_new的文档会新增has_diff和diff_details字段，直接标记差异情况。

内容的提问来源于stack exchange，提问作者young_minds1