Logstash配置遇ConfigurationError，Grok模式匹配报错求助

问题背景 #

我有一个可正常匹配日志的Grok模式，将其配置到Logstash的filter后触发语法错误，尝试转义引号仍未解决。

原Grok模式：

"%{TIMESTAMP_ISO8601:timestamp}"\|"%{DATA:tz}"\|"%{GREEDYDATA:trans}\: %{GREEDYDATA:transId}"\|"%{GREEDYDATA:req}\: %{GREEDYDATA:reqId}"\|"%{IP:ip}"\|"%{GREEDYDATA:path}\=%{GREEDYDATA:codF}"\|"%{DATA:httpver}"\|"%{DATA:app}"\|"%{WORD:verb}"\|"%{GREEDYDATA:gw}\: %{GREEDYDATA:gw_status}"\|"%{GREEDYDATA:be}\: %{GREEDYDATA:be_status}"\|"%{DATA:unknown}"\|"%{DATA:postman}"\|"%{DATA:link}"\|"%{GREEDYDATA:tok}\: %{GREEDYDATA:token}"

配置到Logstash的filter代码：

filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}"\|"%{DATA:tz}"\|"%{GREEDYDATA:trans}\: %{GREEDYDATA:transId}"\|"%{GREEDYDATA:req}\: %{GREEDYDATA:reqId}"\|"%{IP:ip}"\|"%{GREEDYDATA:path}\=%{GREEDYDATA:codF}"\|"%{DATA:httpver}"\|"%{DATA:app}"\|"%{WORD:verb}"\|"%{GREEDYDATA:gw}\: %{GREEDYDATA:gw_status}"\|"%{GREEDYDATA:be}\: %{GREEDYDATA:be_status}"\|"%{DATA:unknown}"\|"%{DATA:postman}"\|"%{DATA:link}"\|"%{GREEDYDATA:tok}\: %{GREEDYDATA:token}" }
    add_field => [ "grok_state", "match" ]
  }
}

错误信息 #

Failed to execute action
{:action=>LogStash::PipelineAction::Create/pipeline_id:main,
:exception=>"LogStash::ConfigurationError", :message=>"Expected one of
[ \t\r\n], "#", "{", "}" at line 10, column 61 (byte 158)
after filter {\n grok {\n match => { "message" =>
"%{TIMESTAMP_ISO8601:timestamp}""",
:backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in
compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:182:in initialize'",
"org/logstash/execution/JavaBasePipelineExt.java:72:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in initialize'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50:in
execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:386:in block
in converge_state'" ]}

日志示例 #

"2022-11-28 09:14:59:514"|"+0100"|"transId: xxx"|"reqId: xxx"|"1.1.1.1"|"/path/codF=xxxxxxxxxxx"|"HTTP/1.1"|"SAP"|"GET"|"gateway status: 200"|"backend status: 200"|""|"Runtime/7.29.2"|"client"|"token: xxxx-xxxx-xxxx"

解决思路及正确配置 #

错误核心是Logstash配置字符串的引号转义规则未正确遵守，同时无需转义管道符。

关键修正点 #

1. 转义字符串内的双引号：整个Grok模式被双引号包裹，模式内需要匹配日志中的双引号时，必须用\"替代"，避免Logstash解析时误判字符串结束位置。
2. 移除管道符的转义：日志中的|是普通分隔符，不需要用\|转义，直接保留原字符即可。

正确的Logstash Filter配置 #

filter {
  grok {
    match => { 
      "message" => "\"%{TIMESTAMP_ISO8601:timestamp}\"|\"%{DATA:tz}\"|\"%{GREEDYDATA:trans}: %{GREEDYDATA:transId}\"|\"%{GREEDYDATA:req}: %{GREEDYDATA:reqId}\"|\"%{IP:ip}\"|\"%{GREEDYDATA:path}=%{GREEDYDATA:codF}\"|\"%{DATA:httpver}\"|\"%{DATA:app}\"|\"%{WORD:verb}\"|\"%{GREEDYDATA:gw}: %{GREEDYDATA:gw_status}\"|\"%{GREEDYDATA:be}: %{GREEDYDATA:be_status}\"|\"%{DATA:unknown}\"|\"%{DATA:postman}\"|\"%{DATA:link}\"|\"%{GREEDYDATA:tok}: %{GREEDYDATA:token}\"" 
    }
    add_field => [ "grok_state", "match" ]
  }
}

额外优化建议 #

• 将Grok模式单独存入patterns目录的自定义文件，通过patterns_dir引用，提升配置可读性与维护性。
• 用logstash -f your_config.conf --config.test_and_exit快速验证配置语法正确性。

内容的提问来源于stack exchange，提问作者stecog