使用Java Rest Client可靠获取Elasticsearch索引所有条目的方法咨询

You’re spot on about the problems with from/size pagination—it’s unreliable for large datasets when documents are being added or deleted, and it hits hard limits (like the default max_result_window of 10,000) that make fetching all results impossible. Luckily, Elasticsearch has built-in tools to stream results safely without loading everything into memory at once.

Use the Scroll API (Classic Approach) #

The Scroll API creates a point-in-time snapshot of your index, so subsequent requests won’t be affected by changes to the data while you’re fetching results. It lets you pull batches of documents incrementally, which is perfect for avoiding memory overflow.

Here’s how to implement it with the Java High Level Rest Client:

// 1. Define your base search request
SearchRequest searchRequest = new SearchRequest("your_index");
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
sourceBuilder.query(QueryBuilders.matchAllQuery());
sourceBuilder.size(1000); // Adjust batch size based on your memory limits
searchRequest.source(sourceBuilder);

// 2. Initialize the scroll with a timeout (how long the snapshot is kept alive)
searchRequest.scroll(TimeValue.timeValueMinutes(1));

// 3. Execute the initial search
SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
String scrollId = searchResponse.getScrollId();
SearchHit[] searchHits = searchResponse.getHits().getHits();

// 4. Stream results in batches
try {
    while (searchHits != null && searchHits.length > 0) {
        // Process each batch of hits here
        for (SearchHit hit : searchHits) {
            // Handle your document data
            String sourceAsString = hit.getSourceAsString();
            // ... your processing logic
        }

        // Prepare the next scroll request
        ScrollScrollRequest scrollRequest = new ScrollScrollRequest(scrollId);
        scrollRequest.scroll(TimeValue.timeValueMinutes(1));

        // Execute the scroll request
        searchResponse = client.scroll(scrollRequest, RequestOptions.DEFAULT);
        scrollId = searchResponse.getScrollId();
        searchHits = searchResponse.getHits().getHits();
    }
} finally {
    // 5. Clean up the scroll context to free resources
    ClearScrollRequest clearScrollRequest = new ClearScrollRequest();
    clearScrollRequest.addScrollId(scrollId);
    client.clearScroll(clearScrollRequest, RequestOptions.DEFAULT);
}

Key Notes for Scroll API: #

• Batch Size: Set size to a value that balances speed and memory usage (1000-5000 is typical, depending on your document size).
• Scroll Timeout: Choose a timeout that’s long enough to process each batch, but not unnecessarily long (it keeps resources tied up on the server).
• Cleanup: Always call clearScroll when you’re done—even if you exit early—to avoid leaving orphaned search contexts on the Elasticsearch cluster.

Point in Time (PIT) + Scroll (Recommended for Elasticsearch 7.10+) #

For more flexibility (especially if you need to keep the snapshot alive longer or across multiple indices), use Point in Time (PIT) combined with Scroll. PIT creates a persistent snapshot that’s independent of the initial search request.

Example workflow:

// 1. Create a Point in Time
OpenPointInTimeRequest pitRequest = new OpenPointInTimeRequest("your_index");
pitRequest.keepAlive(TimeValue.timeValueMinutes(5));
OpenPointInTimeResponse pitResponse = client.openPointInTime(pitRequest, RequestOptions.DEFAULT);
String pitId = pitResponse.getId();

try {
    // 2. Initial scroll request with PIT
    SearchRequest searchRequest = new SearchRequest();
    searchRequest.pointInTime(new PointInTimeBuilder(pitId));
    SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
    sourceBuilder.query(QueryBuilders.matchAllQuery());
    sourceBuilder.size(1000);
    searchRequest.source(sourceBuilder);
    searchRequest.scroll(TimeValue.timeValueMinutes(1));

    SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
    String scrollId = searchResponse.getScrollId();
    SearchHit[] searchHits = searchResponse.getHits().getHits();

    // 3. Stream results
    while (searchHits != null && searchHits.length > 0) {
        // Process hits...

        // Next scroll request
        ScrollScrollRequest scrollRequest = new ScrollScrollRequest(scrollId);
        scrollRequest.scroll(TimeValue.timeValueMinutes(1));
        searchResponse = client.scroll(scrollRequest, RequestOptions.DEFAULT);
        scrollId = searchResponse.getScrollId();
        searchHits = searchResponse.getHits().getHits();
    }
} finally {
    // 4. Close the PIT
    ClosePointInTimeRequest closePitRequest = new ClosePointInTimeRequest(pitId);
    client.closePointInTime(closePitRequest, RequestOptions.DEFAULT);
}

Why PIT is Better: #

• PITs can be reused across multiple search requests, unlike Scroll’s context which is tied to the initial search.
• You can extend the PIT’s keep-alive timeout if needed, without restarting the scroll.

Avoid These Pitfalls #

• Don’t use from/size for large datasets—it’s not designed for it, and you’ll hit the max_result_window limit.
• Never leave Scroll contexts or PITs open indefinitely—they consume cluster resources. Always clean them up in a finally block.
• If you need real-time results (not a snapshot), consider the search_after API instead—but it doesn’t give you a complete point-in-time view, so results can change between requests.

内容的提问来源于stack exchange，提问作者fedd