Logstash-5.6.0无法向Elasticsearch-6.2.1传输解析日志排查求助

Alright, let's break down what might be going wrong here—you've got two linked problems: log parsing failures, and Logstash not sending data to Elasticsearch. Let's tackle them step by step, starting with parsing since that's often the root cause (badly parsed logs might get dropped, or rejected by Elasticsearch outright).

1. Fix Log Parsing Failures First #

Parsing errors are almost always tied to mismatched filter configurations or unhandled log formats. Here's how to diagnose and fix this:

• Check Logstash's debug logs: The most critical step. Look at Logstash's main log file (usually /var/log/logstash/logstash-plain.log on Linux, or logs/logstash-plain.log in your Logstash install directory on Windows). You'll see specific error messages like grokparsefailure or jsonparsefailure that tell you exactly which filter is failing.
• Validate your filter patterns:
  • If you're using grok to parse unstructured logs, test your pattern against your sample log. Use Logstash's built-in test command: run bin/logstash -f logstash.conf --config.test_and_exit to catch syntax errors, and manually align your grok pattern to your log's structure—e.g., if your log starts with [2024-05-20 10:00:00] ERROR: Something broke, your pattern should capture the timestamp, level, and message correctly.
  • If you're parsing JSON logs, ensure your json { source => "message" } filter points to the right field, and confirm your log is valid JSON (even a missing comma will cause a parse failure).
• Handle non-standard encodings: If your logs use a charset other than UTF-8 (like GBK), add a codec to your input block: codec => plain { charset => "GBK" } to avoid garbled text that breaks parsing.

2. Verify Elasticsearch Delivery Configuration #

Once parsing is fixed, make sure Logstash can talk to Elasticsearch properly:

• Check output block syntax: Ensure your Elasticsearch output has the correct basic settings:

  output {
    elasticsearch {
      hosts => ["localhost:9200"] # Match your ES host/port
      index => "your-log-index-%{+YYYY.MM.dd}" # Use a valid index pattern
      # Add these if ES has authentication enabled
      # user => "elastic"
      # password => "your-es-password"
      # Add ssl => true if ES uses HTTPS
    }
  }
  

• Test Elasticsearch connectivity: From the machine running Logstash, run curl http://localhost:9200 (or Invoke-RestMethod http://localhost:9200 on PowerShell). If you don't get a valid ES response, ES isn't running, or there's a network/port issue.
• Check index permissions: Ensure Elasticsearch allows automatic index creation (enabled by default in ES 7+, but if you've disabled it, you'll need to create the index manually first). Also, verify the user Logstash uses has write permissions to the target index.
• Look for delivery errors in Logstash logs: You'll see messages like connection refused (network issue), 401 Unauthorized (bad credentials), or 400 Bad Request (mapping conflict) that point to the exact problem.

3. Validate Your Logstash Startup Command #

• Ensure you're using the right config file: When starting Logstash, use bin/logstash -f /path/to/your/logstash.conf (Linux) or .\bin\logstash.bat -f C:\path\to\logstash.conf (Windows) to explicitly point to your config.
• Test config syntax first: Run bin/logstash -f logstash.conf --config.test_and_exit before starting the service—this will catch typos, missing brackets, or invalid parameters before you even run Logstash.

Quick Final Checks #

• Make sure Logstash and Elasticsearch are running on the same network (or that the ES port is accessible if they're on different machines).
• If you're using ES with security features (like TLS), ensure your Logstash output includes the necessary SSL settings (e.g., ssl => true, cacert => "/path/to/ca.crt").

内容的提问来源于stack exchange，提问作者Karthik Suresh