Logstash配置后ElasticSearch未设置Document ID问题求助

Let's break down why your Elasticsearch _id is still a random string even though the document_id shows up correctly in _source, and how to fix it:

The Root Cause #

By default, Logstash's Elasticsearch output plugin doesn't automatically use the document_id field in your event as the Elasticsearch document's _id. You need to explicitly tell Logstash to map that field to the _id parameter in the output configuration.

Step-by-Step Fix #

1. Locate your Elasticsearch output block in logstash.conf. It probably looks something like this right now:

   output {
     elasticsearch {
       hosts => ["http://127.0.0.1:31311"] # Match your ES port
       index => "twitter_new"
     }
   }
   

2. Add the document_id parameter to reference the field you've extracted. Since you already have the correct value in the document_id field of your event, use the %{field_name} syntax to pass it to Elasticsearch:

   output {
     elasticsearch {
       hosts => ["http://127.0.0.1:31311"]
       index => "twitter_new"
       document_id => "%{document_id}" # This tells Logstash to use the field's value as ES _id
     }
   }
   

3. Verify the field exists (optional but recommended)
   To double-check that your document_id field is correctly populated before it reaches the output, add a stdout output temporarily:

   output {
     stdout { codec => rubydebug } # This will print full event details to Logstash terminal
     elasticsearch {
       hosts => ["http://127.0.0.1:31311"]
       index => "twitter_new"
       document_id => "%{document_id}"
     }
   }
   

   When you send your test request, you should see document_id: 8 in the terminal output.

4. Restart Logstash to apply the new configuration, then re-run your test PowerShell command:

   C:\Users\Me\Downloads\curl-7.64.1-win64-mingw\bin> .\curl.exe -XPUT 'http://127.0.0.1:31311/twitter_new/8'
   

   Now when you query Elasticsearch, the document's _id should match the value you specified (8 in this case).

Key Notes #

• Make sure the field name in %{document_id} exactly matches the field you're using in your Logstash pipeline (check for typos or casing differences).
• If you extracted the ID from the request path (e.g., /twitter_new/8), confirm that your grok or pattern matching is correctly parsing that value into the document_id field.

内容的提问来源于stack exchange，提问作者M.Y. Babt