如何配置ElasticSearch按周期自动删除索引日志？若无可配置项，如何通过Java调用API删除超过1个月的日志

Hey there! Let's break down how to auto-delete logs older than 1 month in Elasticsearch—first using the built-in tooling (the recommended approach), then covering a custom Java API solution if you need more control.

Elasticsearch's ILM is purpose-built for managing time-based data lifecycles like log retention. Here's how to set it up:

Step 1: Create a Lifecycle Policy #

First, define a policy that tells Elasticsearch to delete logs after 30 days. Run this in the Dev Tools console:

PUT _ilm/policy/delete-old-logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "1d" // Optional: Roll over daily if using aliases; skip if you use date-named indices like log-yyyy.MM.dd
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

If you use date-stamped indices (e.g., log-2024.05.20), add these settings to ensure ILM uses the index's date instead of its creation time:

"settings": {
  "index.lifecycle.parse_origination_date": true,
  "index.lifecycle.origination_date_field": "@timestamp" // Optional: Use the log's timestamp instead of index creation date
}

Step 2: Attach the Policy to Your Log Indices #

Link the policy to your log indices via an index template so all new logs inherit it automatically:

PUT _index_template/logs-template
{
  "index_patterns": ["log-*"], // Match your log index pattern
  "template": {
    "settings": {
      "index.lifecycle.name": "delete-old-logs-policy",
      "index.number_of_shards": 1,
      "index.number_of_replicas": 0
    }
  }
}

For existing indices, manually apply the policy:

PUT log-*/_settings
{
  "index.lifecycle.name": "delete-old-logs-policy"
}

Step 3: Verify the Policy Works #

Check policy details:

GET _ilm/policy/delete-old-logs-policy

Or confirm lifecycle status for your indices:

GET log-*/_ilm/explain

If ILM doesn't fit your use case (e.g., you need custom filtering beyond just age), use the Elasticsearch High Level REST Client to delete old logs programmatically.

Step 1: Add Dependencies #

Ensure your client version matches your Elasticsearch cluster (Maven example):

<dependency>
  <groupId>org.elasticsearch.client</groupId>
  <artifactId>elasticsearch-rest-high-level-client</artifactId>
  <version>your-es-version</version>
</dependency>
<dependency>
  <groupId>org.elasticsearch</groupId>
  <artifactId>elasticsearch</artifactId>
  <version>your-es-version</version>
  <scope>provided</scope>
</dependency>

Step 2: Write the Java Code #

Here's a complete example to delete documents older than 30 days:

import org.elasticsearch.action.delete.DeleteByQueryRequest;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.index.query.RangeQueryBuilder;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;

import java.io.IOException;
import java.time.Instant;
import java.time.temporal.ChronoUnit;

public class OldLogDeleter {

    public static void main(String[] args) {
        // Initialize client (adjust host/port to your cluster)
        RestClientBuilder builder = RestClient.builder(
                new org.apache.http.HttpHost("localhost", 9200, "http"));
        try (RestHighLevelClient client = new RestHighLevelClient(builder)) {

            // Target all log indices
            DeleteByQueryRequest request = new DeleteByQueryRequest("log-*");

            // Filter documents older than 30 days (adjust field name if your logs use a different timestamp field)
            Instant thirtyDaysAgo = Instant.now().minus(30, ChronoUnit.DAYS);
            RangeQueryBuilder ageFilter = QueryBuilders.rangeQuery("@timestamp")
                    .lt(thirtyDaysAgo.toEpochMilli());

            request.setQuery(ageFilter);
            request.setRefresh(true); // Refresh indices after deletion
            request.setBatchSize(1000); // Control batch size for performance

            // Execute deletion
            var response = client.deleteByQuery(request, RequestOptions.DEFAULT);
            System.out.println("Deleted " + response.getDeleted() + " old log documents");

        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

Step 3: Schedule the Task #

To run this automatically, use a scheduler like:

• Spring's @Scheduled annotation (for Spring Boot apps)
• Quartz Scheduler (for standalone applications)

Pro Tips: #

• Ensure your client has delete permissions on the log indices.
• For large datasets, split deletions into smaller chunks to avoid performance impacts.
• Test the filter query with a SearchRequest first to verify which documents will be deleted.

内容的提问来源于stack exchange，提问作者java dev