Elasticsearch runtime_mapping lookup类型排序失败问题排查

我尝试对通过runtime_mapping的「lookup」类型从关联索引获取的值进行排序，移除排序条件时能正常返回该字段值，但直接排序或使用多种Painless脚本排序均失败。想确认是v8.2新增的这个功能尚未成熟，还是我的操作有误？

父索引结构 #

{
    "oir_pr_lh_2021": {
        "mappings": {
            "properties": {
                "subToiId": {
                    "type": "integer"
                }
            }
       }
   }
}

Lookup索引结构 #

{
    "sub_toi_lk": {
        "mappings": {
            "properties": {
                "id": {
                    "type": "integer"
                },
                "name": {
                    "type": "keyword"
                }
            }
        }
    }
}

可正常返回Lookup值的查询语句 #

{
    "runtime_mappings": {
        "subToiLk": {
            "type": "lookup",
            "target_index": "sub_toi_lk",
            "input_field": "subToiId",
            "target_field": "id",
            "fetch_fields": [
                "name"
            ]
        }
    },
    "fields": [
        "subToiId",
        "subToiLk"
    ],
    "_source": false
}

部分查询结果 #

{
  "hits": {
    "total": {
      "value": 14,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "oir_pr_lh_2021",
        "_id": "1423949",
        "_score": 1,
        "fields": {
          "subToiLk": [
            {
              "name": [
                "Individual Medicare Supplement - Medicare Select"
              ]
            }
          ],
          "subToiId": [
            732
          ]
        }
      }
    ]
  }
}

尝试排序的失败案例 #

案例1：脚本访问subToiLk.name排序 #

查询语句：

{
    "runtime_mappings": {
        "subToiLk": {
            "type": "lookup",
            "target_index": "sub_toi_lk",
            "input_field": "subToiId",
            "target_field": "id",
            "fetch_fields": [
                "name"
            ]
        }
    },
    "sort": {
        "_script": {
            "type": "string",
            "script": {
                "lang": "painless",
                "source": "doc['subToiLk.name'].value"
            },
            "order": "asc"
        }
    },
    "fields": [
        "subToiId",
        "subToiLk"
    ],
    "_source": false
}

返回错误：

{
  "shard": 0,
  "index": "oir_pr_lh_2021",
  "node": "ka3YrBmnRCSxjD0eF60KXw",
  "reason": {
    "type": "script_exception",
    "reason": "runtime error",
    "script_stack": [
      "org.elasticsearch.server@8.5.0/org.elasticsearch.search.lookup.LeafDocLookup.getFactoryForDoc(LeafDocLookup.java:127)",
      "org.elasticsearch.server@8.5.0/org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:171)",
      "org.elasticsearch.server@8.5.0/org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:32)",
      "doc['subToiLk.name'].value",
      "    ^---- HERE"
    ],
    "script": "doc['subToiLk.name'].value",
    "lang": "painless",
    "position": {
      "offset": 4,
      "start": 0,
      "end": 26
    },
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "No field found for [subToiLk.name] in mapping"
    }
  }
}

案例2：脚本直接访问subToiLk排序 #

修改后的脚本部分：

{
    "script": {
        "lang": "painless",
        "source": "doc['subToiLk'].value"
    }
}

返回错误：

{
  "shard": 0,
  "index": "oir_pr_lh_2021",
  "node": "ka3YrBmnRCSxjD0eF60KXw",
  "reason": {
    "type": "script_exception",
    "reason": "runtime error",
    "script_stack": [
      "org.elasticsearch.server@8.5.0/org.elasticsearch.index.mapper.MappedFieldType.fielddataBuilder(MappedFieldType.java:101)",
      "org.elasticsearch.server@8.5.0/org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:102)",
      "org.elasticsearch.server@8.5.0/org.elasticsearch.index.query.SearchExecutionContext.lambda$lookup$2(SearchExecutionContext.java:500)",
      "org.elasticsearch.server@8.5.0/org.elasticsearch.search.lookup.SearchLookup.getForField(SearchLookup.java:114)",
      "org.elasticsearch.server@8.5.0/org.elasticsearch.search.lookup.LeafDocLookup$2.run(LeafDocLookup.java:136)",
      "org.elasticsearch.server@8.5.0/org.elasticsearch.search.lookup.LeafDocLookup$2.run(LeafDocLookup.java:132)",
      "java.base/java.security.AccessController.doPrivileged(AccessController.java:318)",
      "org.elasticsearch.server@8.5.0/org.elasticsearch.search.lookup.LeafDocLookup.getFactoryForDoc(LeafDocLookup.java:132)",
      "org.elasticsearch.server@8.5.0/org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:171)",
      "org.elasticsearch.server@8.5.0/org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:32)",
      "doc['subToiLk'].value",
      "    ^---- HERE"
    ],
    "script": "doc['subToiLk'].value",
    "lang": "painless",
    "position": {
      "offset": 4,
      "start": 0,
      "end": 21
    },
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "Fielddata is not supported on field [subToiLk] of type [lookup]"
    }
  }
}

案例3：直接使用字段路径排序 #

查询语句中的sort部分：

{
    "sort": [
            "subToiLk.name"
    ]
}

返回错误：

{
    "error": {
        "root_cause": [
            {
                "type": "query_shard_exception",
                "reason": "No mapping found for [subToiLk.name] in order to sort on",
                "index_uuid": "eKAO5aYJQuKMK3t1ZaOPcQ",
                "index": "oir_pr_lh_2021"
            }
        ]
    }
}

问题解答 #

这不是操作错误，而是当前Elasticsearch的Runtime Lookup字段不支持排序，该功能存在设计局限性：

1. 从错误信息Fielddata is not supported on field [subToiLk] of type [lookup]可明确，Lookup类型的runtime字段不支持fielddata，而排序依赖fielddata加载字段值到内存进行计算。
2. Lookup字段是查询阶段动态从关联索引获取数据的，未预加载或存储在主索引中，无法提供排序所需的底层支持。
3. 官方文档明确说明，Lookup类型的runtime字段不支持排序、聚合、doc上下文的脚本访问，仅能作为查询结果返回。

替代方案 #

如果需要实现基于关联字段的排序，可考虑两种方式：

• 预关联数据：写入主索引时，将lookup索引中的name字段同步写入主索引作为普通keyword字段，之后直接对该字段排序。
• 使用Join类型：将主索引与lookup索引设置为Join关系，通过has_child或has_parent查询结合排序，但性能可能不如预关联数据。

内容的提问来源于stack exchange，提问作者john nowlin