Logstash多管道启动偶发StringIndexOutOfBoundsException异常求助

在Windows主机部署ELK栈的Logstash，负责读取本地日志并发送至远端Elasticsearch。采用收集器模式配置3个管道：两个文件处理管道分别处理不同日志文件，再将数据转发到收集器管道统一输出到Elasticsearch。

单独启用收集器管道+任意一个文件处理管道时运行正常，但同时启用两个文件处理管道时，Logstash启动偶发报错（约每5次出现1次），异常始终触发在wms-wmspipeline，错误为StringIndexOutOfBoundsException: String index out of range: -1，涉及组件不定，有时是cabin/outputs/stdlib-logger、grok-pure或cabin/mixins/timer。

已尝试以下排查操作，问题未解决：

• 调整管道启动顺序
• 设置pipeline workers为1
• 注释管道内的过滤器逻辑
• 启用TRACE日志级别排查

完整配置文件 #

pipelines.yml #

- pipeline.id: wms-pdainteractionpipeline
path.config: "config/wms-pdainteraction.conf"

- pipeline.id: wms-wmspipeline
path.config: "config/wms-wms.conf"

- pipeline.id: es
path.config: "config/elasticsearch.conf"

wms-pdainteraction.conf #

input {
file {
path => "C:/wms/pdainteraction.log"
#start_position => "beginning" # ignored when mode=read
codec => plain { charset => "ISO-8859-1" }
}
}

filter {
grok {
match => { "message" => "%{DATESTAMP:log_timestamp} \[%{DATA:java_thread}\] \[%{DATA:pda_aktion}\] %{GREEDYDATA:pda_aktion_parameter} %{DATA:aktion_timestamp} \[%{DATA:benutzer}\]" }
}
date {
match => ["aktion_timestamp", "ISO8601"]
timezone => "UTC"
}
mutate {
add_field => {
"[data_stream][type]" => "logs"
"[data_stream][dataset]" => "pdainteraction"
"[data_stream][namespace]" => "prod"
}
remove_field => ["aktion_timestamp", "log_timestamp", "message"]
}
}

output {
pipeline { send_to => ["es-pipeline"] }
}

wms-wms.conf #

input {
file {
path => "C:/wms/wms.log"
#start_position => "beginning" # ignored when mode=read
mode => "read"
codec => multiline {
charset => "ISO-8859-1"
pattern => "^\d{2}:\d{2}:\d{2}\.\d{3}"
negate => true
what => "previous"
}
}
}

filter {
grok {
match => { "message" => "%{TIME:time} %{LOGLEVEL:loglevel} +\[%{DATA:thread}\] %{DATA:logger}\:%{NUMBER:line} +%{GREEDYDATA:message}" }
add_field => { "date" => "%{+YYYY-MM-dd}"}
add_field => { "timestamp" => "%{date} %{time}" }
overwrite => ["message"]
}

date {
target => "@timestamp"
match => ["timestamp", "YYYY-MM-dd HH:mm:ss.SSS"]
}

mutate {
add_field => {
"[data_stream][type]" => "logs"
"[data_stream][dataset]" => "wms"
"[data_stream][namespace]" => "prod"
}
remove_field => ["date", "time", "timestamp"]
}
}

output {
pipeline {
send_to => ["es-pipeline"]
}
}

elasticsearch.conf #

input {
pipeline {
address => "es-pipeline"
  }
}

output {
elasticsearch {
hosts => ["https://server:9200"]
ssl_enabled => true
ssl_certificate_authorities => '/path/to/ca.crt'
api_key => "REDACTED"
data_stream => "true"
  }
}

完整堆栈跟踪 #

[2024-06-03T13:38:50,166][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:wms-wmspipeline, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (LoadError) load error: cabin/outputs/stdlib-logger -- java.lang.StringIndexOutOfBoundsException: String index out of range: -1", :backtrace=>["org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:186)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1319)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)", "org.jruby.RubyClass.newInstance(RubyClass.java:931)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:548)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:291)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:328)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:144)", "org.jruby.RubyProc.call(RubyProc.java:352)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)", "java.base/java.lang.Thread.run(Thread.java:840)"]}

内容的提问来源于stack exchange，提问作者oxoma