Spring Boot向Logstash传数据时如何避免Elasticsearch重复解析数据？

Hey there! Let's tackle that duplicate data issue you're seeing in Elasticsearch. The root problem here is that Logstash is sending identical (or nearly identical) log entries as new documents to Elasticsearch every time, instead of recognizing they're duplicates and overwriting the existing ones.

The solution relies on generating a unique "fingerprint" for each log entry, then telling Elasticsearch to use that fingerprint as the document ID. This way, if the same log entry comes through again, Elasticsearch will overwrite the existing document instead of creating a new one.

Step 1: Add the Fingerprint Filter to Logstash #

First, you'll need to add the fingerprint filter to your Logstash pipeline. This filter creates a unique hash based on fields that uniquely identify each log entry. Pick fields that don't change between duplicate entries—like the log timestamp, message content, or a unique trace ID (if you're using Spring Cloud Sleuth for distributed tracing).

Here's how to add it to your filter section:

filter {
  # Adjust the source fields to match your actual log structure
  fingerprint {
    source => ["@timestamp", "message", "traceId"]
    target => "[@metadata][fingerprint]"
    method => "SHA256"
    key => "your-unique-secret-key" # Optional: Adds extra uniqueness to the hash
  }

  # Keep your existing filters (like grok, json, etc.) here
}

• source: List of fields that combined make each log entry unique. Tweak this based on your Spring Boot log format.
• target: Stores the fingerprint in Logstash's metadata (so it doesn't get written to Elasticsearch, saving space).
• method: Uses SHA256 for a secure, collision-resistant hash.
• key: Optional, but adding a secret key ensures even if other systems generate similar field values, your fingerprints stay unique.

Step 2: Configure Elasticsearch Output to Use the Fingerprint #

Next, update your Elasticsearch output to use the generated fingerprint as the document_id. This tells Elasticsearch to either create a new document (if the ID doesn't exist) or overwrite the existing one (if it does).

Here's the updated output section:

output {
  elasticsearch {
    hosts => ["http://your-elasticsearch-host:9200"]
    index => "spring-boot-logs-%{+YYYY.MM.dd}" # Keep your existing index pattern
    document_id => "%{[@metadata][fingerprint]}" # Use the fingerprint as the document ID
    action => "update" # Optional: Tells Elasticsearch to update existing docs
    doc_as_upsert => true # Required if using action=update: Creates a new doc if the ID doesn't exist
  }

  # Keep your other outputs (like stdout) here if needed
}

• document_id: This is the critical line—by setting it to the fingerprint, duplicates will overwrite instead of creating new entries.
• action and doc_as_upsert: This combination ensures that if a log entry is new (no matching fingerprint), it gets inserted; if it's a duplicate, it updates the existing document.

Key Notes to Keep in Mind #

• Choose the right source fields: Make sure the fields you use in the fingerprint filter are truly unique per log entry. If you don't have a traceId, you could use a combination of @timestamp, message, and logger_name (if that helps distinguish entries).
• Test first: Before deploying to production, use Logstash's stdout output to verify that the fingerprint is being generated correctly. You can add this to your output section temporarily:

  stdout { codec => rubydebug }
  

• Spring Boot log formatting: If your logs don't include a unique identifier, consider configuring Spring Boot's MDC (Mapped Diagnostic Context) to add a unique request ID or trace ID to each log entry. This will make your fingerprint even more reliable.

内容的提问来源于stack exchange，提问作者Amine Wartateni