Kibana5.6.8配套Logstash仅读取单个日志文件问题求助

Hey there, let's work through why your Logstash setup is only picking up one log file instead of all matching *Web.log* files under your target directory. Here are actionable fixes to test out:

1. Verify Path Wildcards & File Permissions #

Your path pattern /home/elasticsearch/static_logs/**/*Web.log* should recursively match all relevant files, but let's rule out two common issues:

• Test the path directly: Run ls /home/elasticsearch/static_logs/**/*Web.log* as the user Logstash runs under (usually logstash). If this command doesn't list all expected files, adjust directory/file permissions or add the Logstash user to the appropriate group to grant read access.
• Simplify the path first: Try removing the recursive ** temporarily to test with /home/elasticsearch/static_logs/*Web.log*—if this works, the issue might be with nested directory permissions or how Logstash scans subdirectories. You can also add discover_interval => 10 to your file input to make Logstash scan for new files more frequently (default is 15 seconds).

2. Fix Relative Path for Patterns Directory #

Your patterns_dir => "./patterns" uses a relative path, which resolves to the directory where you start Logstash, not your config file's directory (/home/elasticsearch/confLogs). If Logstash can't find your custom patterns, this can break grok parsing and make it seem like files aren't being read. Switch to an absolute path:

patterns_dir => "/home/elasticsearch/confLogs/patterns"

3. Complete & Validate Your Grok Configuration #

Your provided config cuts off mid-grok setup—an incomplete grok pattern will cause configuration errors or failed parsing, which might lead to logs being dropped or unprocessed. For example, if your log lines look like 2017-08-07 11:47:35,466 INFO [http-bio-10.60.2.19-10267-exec-1] ..., you might need a pattern like:

grok {
  patterns_dir => "/home/elasticsearch/confLogs/patterns"
  overwrite => [ "message" ]
  match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} \[%{DATA:thread}\] %{GREEDYDATA:content}" }
}

Always validate your config before running Logstash with:

logstash -f /home/elasticsearch/confLogs/logstash.conf --config.test_and_exit

4. Adjust Sincedb & Scanning Settings #

While setting sincedb_path => "/dev/null" forces Logstash to re-read files on restart, you can tweak scanning behavior to catch new files faster:

input {
  file {
    type => "static"
    path => "/home/elasticsearch/static_logs/**/*Web.log*"
    exclude => "*.zip"
    start_position => beginning
    sincedb_path => "/dev/null"
    stat_interval => 5  # Check file metadata every 5 seconds
    discover_interval => 10  # Scan for new files every 10 seconds
  }
}

5. Check Logstash Logs for Errors #

Don't forget to check Logstash's own logs (typically at /var/log/logstash/logstash.log) for clues. Look for errors about permission denied, missing pattern files, or grok parsing failures—these will point you directly to the root issue.

Your Original Config (For Reference) #

input { 
  file { 
    type => "static" 
    path => "/home/elasticsearch/static_logs/**/*Web.log*" 
    exclude => "*.zip" 
    start_position => beginning 
    sincedb_path => "/dev/null" 
  } 
} 
filter { 
  if [type] == "static" { 
    if [message] !~ /(.+)/ { 
      drop { } 
    } 
    grok{ 
      patterns_dir => "./patterns" 
      overwrite => [ "message" ] 
      # 2017-08-07 11:47:35,466 INFO [http-bio-10.60.2.19-10267-e... 
    } 
  }
}

内容的提问来源于stack exchange，提问作者pcouas