Logstash无法自动读取日志文件并推送至Kibana问题求助

It’s frustrating when Logstash stops tailing logs unless you restart it—let’s walk through the most common fixes for this issue, based on typical configuration pitfalls and known behavior.

1. Fix Sincedb Tracking Issues #

Logstash uses a sincedb file to keep track of how far it’s read each log file. If this file is corrupted, missing, or Logstash can’t write to it, it might think it’s already processed all logs (or fail to update its progress).

• Check sincedb location: On Linux, the default path is /var/lib/logstash/.sincedb (or ~/.sincedb if running as a non-root user). Verify the file exists and the logstash user has write permissions to it.
• Reset sincedb: If you suspect corruption, delete the sincedb file and restart Logstash. Note this will make Logstash reprocess all logs from the start, so only do this if that’s acceptable.
• Explicitly set sincedb path: Add sincedb_path to your file input config to use a dedicated, writable location. Example:

input {
  file {
    path => "/var/log/your-app/*.log"
    sincedb_path => "/var/lib/logstash/sincedb/your-app-sincedb"
    start_position => "beginning" # Only use this initially if you need to backfill logs
  }
}

2. Verify File & Directory Permissions #

Logstash runs under the logstash user (by default on Linux). If this user can’t read your log files or traverse their parent directories, it won’t pick up new logs.

• Run ls -l /var/log/your-app/ to check ownership and permissions. Look for read access (r) for the group or others.
• Add the logstash user to the group that owns your logs (e.g., adm for system logs):

usermod -aG adm logstash

• Restart Logstash after changing permissions to apply the new group membership.

3. Handle Log Rotation Properly #

If your logs are rotated (e.g., with logrotate), Logstash might miss the new files unless configured to detect changes frequently.

• Adjust discovery interval: Use discover_interval to make Logstash check for new files more often (default is 15 seconds). Lower it to 5 seconds for faster detection:

file {
  path => "/var/log/your-app/*.log"
  discover_interval => 5
}

• Avoid copytruncate if possible: This logrotate option copies the log file then truncates it, which can confuse Logstash’s file tracking. If you must use it, enable stat_interval to check file size changes more frequently:

file {
  path => "/var/log/your-app/*.log"
  stat_interval => 2 # Check file stats every 2 seconds
}

4. Fix Input Configuration Mistakes #

Double-check your file input path—small errors here are a common culprit:

• Use absolute paths (not relative ones) to avoid confusion about where Logstash is looking.
• Ensure wildcards match your log files (e.g., *.log if your files end with .log, not .txt).
• Use /**/ to recursively scan subdirectories: /var/log/your-app/**/*.log.

5. Check for Pipeline Bottlenecks or Resource Issues #

If Logstash is stuck processing existing logs or hitting resource limits, it might stop picking up new ones.

• Review Logstash logs: Check /var/log/logstash/logstash-plain.log for errors about pipeline stalls, memory limits, or Elasticsearch output failures.
• Increase JVM heap size: If you see memory-related errors, edit jvm.options (usually in /etc/logstash/) to raise -Xms and -Xmx (e.g., -Xms4g -Xmx4g for 4GB of heap).
• Verify Elasticsearch connectivity: If Elasticsearch is down or slow, Logstash will back up and stop processing new logs until the output is available. Test connectivity with curl http://your-es-host:9200.

6. Consider Using Filebeat (More Reliable Log Collection) #

If you keep running into issues with Logstash’s file input, switch to Filebeat for log collection. Filebeat is purpose-built for tailing logs and handles rotation, network interruptions, and file tracking more reliably than Logstash’s input plugin.

Example Filebeat config:

filebeat.inputs:
- type: log
  paths:
    - /var/log/your-app/*.log

output.logstash:
  hosts: ["localhost:5044"]

Corresponding Logstash input:

input {
  beats {
    port => 5044
  }
}

After trying these steps, test by adding a new log entry and see if Logstash picks it up without restarting. If the issue persists, share your full Logstash config and relevant lines from the Logstash logs so we can troubleshoot further.

内容的提问来源于stack exchange，提问作者Mikey R