Logstash从7.17.9升级到7.17.10后出现管道错误

本周初完成Logstash从7.17.9到7.17.10的升级后，日志无法同步至Kibana。经初步排查，问题出在Logstash服务器上：服务启动约30秒后就会出现管道工作线程错误，随后崩溃，未监听5000端口，导致Filebeat无法建立连接。

Logstash管道工作线程错误信息 #

[2023-05-11T12:10:07,387][ERROR][logstash.javapipeline    ][main] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"main", :error=>"(NoMethodError) undefined method `shutdown_requested' for #<LogStash::JavaPipeline:0x60319724>", :exception=>Java::OrgJrubyExceptions::NoMethodError, :backtrace=>["usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_11_dot_15_dot_2_minus_java.lib.logstash.plugin_mixins.elasticsearch.common.pipeline_shutdown_requested?(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.15.2-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:380)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_11_dot_15_dot_2_minus_java.lib.logstash.outputs.elasticsearch.wait_for_successful_connection(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.15.2-java/lib/logstash/outputs/elasticsearch.rb:440)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_11_dot_15_dot_2_minus_java.lib.logstash.outputs.elasticsearch.multi_receive(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.15.2-java/lib/logstash/outputs/elasticsearch.rb:390)", "org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.multi_receive(org/logstash/config/ir/compiler/OutputStrategyExt.java:143)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.multi_receive(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:300)"], :thread=>"#<Thread:0x42c34b77 sleep>"}

核心错误点：(NoMethodError) undefined method 'shutdown_requested' for #<LogStash::JavaPipeline:0x60319724>

Filebeat连接错误信息 #

May 11 06:12:41 logs01.mgmt filebeat[18266]: 2023-05-11T06:12:41.005-0600        ERROR        [publisher_pipeline_output]        pipeline/output.go:154        Failed to connect to failover(backoff(async(tcp://10.0.0.226:5000)),backoff(async(tcp://10.0.0.227:5000))): dial tcp 10.0.0.226:5000: connect: connection refused
May 11 06:12:41 logs01.mgmt filebeat[18266]: 2023-05-11T06:12:41.005-0600        INFO        [publisher_pipeline_output]        pipeline/output.go:145        Attempting to reconnect to failover(backoff(async(tcp://10.0.0.226:5000)),backoff(async(tcp://10.0.0.227:5000))) with 26 reconnect attempt(s)

相关配置 #

Filebeat配置 #

output.logstash:
  # The Logstash hosts
  hosts: ["10.0.0.226:5000","10.0.0.227:5000"]

Logstash输入配置 #

input {
  beats {
    port => 5000
    type => syslog
  }
}

已完成的排查操作 #

• 确认Logstash服务器未监听5000端口，服务启动后随即崩溃
• 验证两端服务器无防火墙运行
• 确认SELinux已禁用
• 以root身份重启Logstash和Filebeat，问题仍未解决

内容的提问来源于stack exchange，提问作者Gabriel Xerxes