Spring从Elasticsearch获取超1万条数据过慢的优化方案咨询

问题场景 #

我有一个包含2万余条文档的Elasticsearch索引，最初使用以下代码获取数据：

final SearchHits<MyTyp> searchHitsMyTyp =
        elasticsearchCustomTemplate.search(
                searchQueryMyTyp,
                MyTyp.class,
                IndexCoordinates.of(indexSettings.indexMyTypName))

这段代码能立即返回1万条文档，但无法获取全部数据，因此修改为基于search_after的分页实现来获取全量数据：

NativeSearchQuery searchQuery =
        new NativeSearchQueryBuilder()
                .withQuery(
                        QueryBuilders.matchQuery(
                                "MyTyp_id", mytyp.getId()))
                .build();
                
NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder();
queryBuilder.withQuery(QueryBuilders.termQuery("MyTyp_id", mytyp.getId()));
queryBuilder.withPageable(PageRequest.of(0, 10000));
queryBuilder.withSort(SortBuilders.fieldSort("_id").order(SortOrder.ASC));

queryBuilder.withAggregations(
        AggregationBuilders.filter(
                "filter_by_id", QueryBuilders.termQuery("MyTyp_id", mytyp.getId())));
NativeSearchQuery searchQueryMyTyp = queryBuilder.build();
SearchHit[] searchHitsMyTyp=null;
final ArrayList<MyTyp> typeList=new ArrayList();

do {
  if(searchHitsMyTyp!=null) {
    searchQuery.setSearchAfter(searchHitsMyTyp[searchHitsMyTyp.length - 1].getSortValues().stream().toList());
  }
  searchHitsMyTyp =  elasticsearchCustomTemplate.search(
          searchQueryMyTyp,
          MyTyp.class,
          IndexCoordinates.of(indexSettings.indexMyTypName)).getSearchHits().toArray(new SearchHit[0]);
  typeList.addAll( Arrays.stream(searchHitsMyTyp).map(p->(MyTyp)p.getContent()).toList());

}while(searchHitsMyTyp.length >= 10000);

现在的问题是该实现速度极慢，导致UI卡顿。按1万条耗时1秒计算，2万条理论上约2秒，但实际远超这个时间，想知道是否有优化方法，或是Elasticsearch与Spring的固有限制？

优化方案 #

1. 移除冗余聚合操作 #

代码中添加了和查询条件重复的filter聚合，但实际并未使用聚合结果，该操作会额外消耗ES计算资源，直接删除：

// 删除这段无用的聚合代码
// queryBuilder.withAggregations(
//        AggregationBuilders.filter(
//                "filter_by_id", QueryBuilders.termQuery("MyTyp_id", mytyp.getId())));

2. 修正search_after逻辑错误 #

代码中存在逻辑漏洞：初始化searchQuery后，循环中始终使用的是未更新search_after的searchQueryMyTyp，导致每次查询都从第一条开始。正确做法是在循环中更新查询的search_after参数：

NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder();
queryBuilder.withQuery(QueryBuilders.termQuery("MyTyp_id", mytyp.getId()));
queryBuilder.withPageable(PageRequest.of(0, 10000));
queryBuilder.withSort(SortBuilders.fieldSort("_id").order(SortOrder.ASC));

NativeSearchQuery searchQueryMyTyp = queryBuilder.build();
SearchHit[] searchHitsMyTyp = null;
final ArrayList<MyTyp> typeList = new ArrayList();

do {
    if (searchHitsMyTyp != null) {
        // 更新当前查询的search_after参数
        queryBuilder.withSearchAfter(searchHitsMyTyp[searchHitsMyTyp.length - 1].getSortValues());
        searchQueryMyTyp = queryBuilder.build();
    }
    SearchHits<MyTyp> hits = elasticsearchCustomTemplate.search(
            searchQueryMyTyp,
            MyTyp.class,
            IndexCoordinates.of(indexSettings.indexMyTypName));
    searchHitsMyTyp = hits.getSearchHits().toArray(new SearchHit[0]);
    typeList.addAll(Arrays.stream(searchHitsMyTyp).map(p -> p.getContent()).toList());

} while (searchHitsMyTyp.length >= 10000);

3. 异步化查询避免UI阻塞 #

当前同步执行的查询会阻塞UI线程，可将查询逻辑放到异步线程中执行，比如用Spring的@Async注解或手动线程池：

// 示例：用CompletableFuture异步执行
CompletableFuture.supplyAsync(() -> {
    // 这里放你的查询逻辑
    return typeList;
}, executorService).thenAccept(result -> {
    // 查询完成后通知UI更新数据
});

4. 替换为Scroll API（全量获取场景） #

如果是一次性全量获取数据，ES的Scroll API比search_after更稳定，它会创建数据快照，避免后续数据变更影响结果：

// 初始化Scroll查询
NativeSearchQuery scrollQuery = new NativeSearchQueryBuilder()
        .withQuery(QueryBuilders.termQuery("MyTyp_id", mytyp.getId()))
        .withPageable(PageRequest.of(0, 10000))
        .build();

ScrollHits<MyTyp> scrollHits = elasticsearchCustomTemplate.scroll(1000L, scrollQuery, MyTyp.class, IndexCoordinates.of(indexSettings.indexMyTypName));
String scrollId = scrollHits.getScrollId();
List<MyTyp> typeList = new ArrayList<>();

while (scrollHits.hasSearchHits()) {
    typeList.addAll(scrollHits.getSearchHits().stream().map(SearchHit::getContent).toList());
    scrollHits = elasticsearchCustomTemplate.scrollScrollId(1000L, scrollId, MyTyp.class, IndexCoordinates.of(indexSettings.indexMyTypName));
    scrollId = scrollHits.getScrollId();
}

// 清理Scroll上下文，避免资源泄漏
elasticsearchCustomTemplate.clearScroll(scrollId);

5. 字段过滤减少数据传输 #

如果不需要MyTyp的所有字段，可通过fetchSource指定仅返回需要的字段，降低ES到应用的数据传输量：

// 只返回field1和field2字段
queryBuilder.withSourceFilter(new FetchSourceFilter(new String[]{"field1", "field2"}, null));

关于固有限制 #

Elasticsearch和Spring Data Elasticsearch本身没有针对2万条数据的性能限制，你的问题主要来自代码逻辑错误、冗余操作以及同步阻塞UI的执行方式，优化后可以达到预期的性能。

内容的提问来源于stack exchange，提问作者user2557930