使用systemctl启动Logstash服务后日志无法同步至Kibana求助

我之前碰到过几乎一模一样的场景，先从几个最常见的排查方向入手，应该能解决你的问题：

1. 文件权限不匹配（最可能的原因） #

当你用sudo systemctl start logstash启动时，服务是用logstash用户运行的；而命令行直接执行bin/logstash -f ...时，是用当前登录用户（比如ubuntu）运行的。你的输入日志路径是/home/ubuntu/log/*，很可能logstash用户没有访问这个目录或文件的权限。

检查方法： #

执行以下命令查看权限：

ls -ld /home/ubuntu/log
ls -l /home/ubuntu/log/

如果logstash用户没有读权限，你会看到目录/文件的权限列表里没有r权限给other或者logstash用户/组。

解决方法： #

• 给logstash用户添加目录的读权限：

sudo setfacl -R -m u:logstash:r /home/ubuntu/log

• 或者把logstash用户加入ubuntu用户组（如果日志目录属于ubuntu组）：

sudo usermod -aG ubuntu logstash
sudo systemctl restart logstash

2. Sincedb文件导致日志被标记为已处理 #

Logstash的file输入会用sincedb文件记录已经读取过的日志位置，命令行启动和systemd启动使用的sincedb路径不同：

• 命令行启动时，sincedb默认在当前用户的主目录下（比如~/.sincedb_xxxx）
• systemd启动时，从你的日志里能看到sincedb路径是/var/lib/logstash/plugins/inputs/file/.sincedb_xxx

所以当你用命令行测试后，日志已经被标记为已读取，切换到systemd启动时，Logstash会认为这些日志已经处理过，不会再读取。

解决方法： #

• 删除systemd启动生成的sincedb文件：

sudo rm /var/lib/logstash/plugins/inputs/file/.sincedb_*
sudo systemctl restart logstash

• 或者在logstash.conf的file输入块里手动指定统一的sincedb路径，确保两种启动方式共用同一个文件：

input {
  file {
    path => "/home/ubuntu/log/*"
    start_position => "beginning"
    sincedb_path => "/var/lib/logstash/plugins/inputs/file/.sincedb"
  }
}

然后确保logstash用户对该路径有读写权限。

3. 验证配置是否正确加载 #

有时候systemd启动时可能没有加载到你预期的配置文件，比如/etc/logstash/conf.d/logstash.conf。

检查方法： #

• 查看Logstash的完整日志文件/var/log/logstash/logstash.log，搜索是否有加载配置的相关日志，确认是否包含你的conf文件。
• 用logstash用户身份测试配置语法和加载情况：

su - logstash -c '/usr/share/logstash/bin/logstash --path.settings /etc/logstash -t'

如果配置有问题，这里会输出错误信息。

4. 环境变量差异 #

命令行启动和systemd启动的环境变量可能存在差异（比如JAVA_HOME、内存参数等），可能影响Logstash的运行。

检查方法： #

• 对比命令行下的环境变量和systemd服务的环境变量：
  • 命令行执行env | grep -E "(JAVA_HOME|LOGSTASH)"
  • 查看systemd服务的环境文件：cat /etc/default/logstash 和 cat /etc/sysconfig/logstash
• 如果发现差异，调整systemd的环境文件使其与命令行一致，然后重启服务。

附你提供的相关信息： #

Logstash Service Status Log

● logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: enabled)
Active: active (running) since Thu 2020-07-09 11:51:32 UTC; 51min ago
Main PID: 20795 (java)
Tasks: 49 (limit: 4915)
CGroup: /system.slice/logstash.service
└─20795 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit
Jul 09 11:51:56 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:56,406][WARN ][logstash.outputs.amazonelasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_versi
Jul 09 11:51:56 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:56,431][INFO ][logstash.outputs.amazonelasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::AmazonElasticSearch", :hosts=>["https://vpc-logerror-vilj5v
Jul 09 11:51:56 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:56,457][INFO ][logstash.outputs.amazonelasticsearch][main] Using mapping template from {:path=>nil}
Jul 09 11:51:56 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:56,497][INFO ][logstash.outputs.amazonelasticsearch][main] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60002, "settings"=>{"inde
Jul 09 11:51:56 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:56,548][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pip
Jul 09 11:51:58 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:58,041][INFO ][logstash.inputs.file ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sinc
Jul 09 11:51:58 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:58,086][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
Jul 09 11:51:58 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:58,244][INFO ][filewatch.observingtail ][main][08f9a378b558a96db1555616c42d9ed88ef9b26655c2be176bc35c226b2e5572] START, creating Discoverer, Watch with file and sincedb co
Jul 09 11:51:58 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:58,248][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
Jul 09 11:51:58 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:58,718][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

Configuration Files

logstash.service

[Unit]
Description=logstash
[Service]
Type=simple
User=logstash
Group=logstash
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384
TimeoutStopSec=infinity
[Install]
WantedBy=multi-user.target

logstash.yml

path.data: /var/lib/logstash
pipeline.ordered: auto
path.logs: /var/log/logstash

pipeline.yml

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"

logstash.conf

input {
  file {
    path => "/home/ubuntu/log/*"
    start_position => "beginning"
  }
}
filter {
  grok {
    match => { "message" => "(?<jsonf>({.*}))"}
  }
  json {
    source => "jsonf"
  }
  mutate {
    remove_field => [ "message","jsonf" ]
  }
}
output {
  amazon_es {
    hosts => ["https://*****************.es.amazonaws.com"]
    region => "us-east-1"
    index => "errorlogs-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}

内容的提问来源于stack exchange，提问作者Rahul Ravichandran