Elasticsearch Grok管道测试正常但FileBeat使用时无日志显示求助

我创建了名为ExtractOwaspErrorFields的Ingest Pipeline，其中包含Grok处理器用于提取message字段内容。该管道在Kibana的Stack Management→Ingest Pipelines→Test Pipeline功能中，使用指定_id和_index的真实日志文档测试时完全正常，能正确提取字段。但将该管道配置到filebeat.yml的output.elasticsearch.pipeline后，Kibana的Discovery中完全看不到任何文档。

相关配置信息 #

filebeat.yml配置 #

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  username: xxx
  password: xxx
  pipeline: ExtractOwaspErrorFields

管道配置 #

{
  "ExtractOwaspErrorFields" : {
    "processors" : [
      {
        "grok" : {
          "field" : "message",
          "patterns" : [
            "%{OWASP_ERRORLOG}"
          ],
          "pattern_definitions" : {
            "OWASP_ERRORLOG" : "\[%{HTTPDERROR_DATE:timestamp}\] \[:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}:tid %{NUMBER:tid}\]?( \[client %{IPORHOST:client}:%{POSINT:clientport}\]) \[client %{IPORHOST:client_ip}\] ModSecurity: ?(%{APACHE_ERROR_MESSAGE:error}) \[file \"%{PATH:matching_rule_file}\"\] \[line \"%{POSINT:matching_rule_line}\"\] \[id \"%{NUMBER:matching_rule_id}\"\] \[msg \"%{DATA:owasp_message_string}\"\] \[data \"%{DATA:owasp_message_data}\"\] \[severity \"%{WORD:owasp_severity}\"\] \[ver \"%{DATA:owasp_version}\"\] %{GREEDYDATA:tags} \[hostname \"%{HOSTNAME:hostname}\"\] \[uri \"%{URIPATHPARAM:uri}\"\] \[unique_id \"%{DATA:unique_id}\"\]",
            "APACHE_ERROR_MESSAGE" : "( .+?(?= \[%{WORD} \"))"
          },
          "if" : "ctx?.docker.container.labels.com_docker_stack_namespace == 'modsecurity'",
          "ignore_failure" : true
        }
      }
    ]
  }
}

测试用日志文档 #

{
  "docs":
  [
    {
      "_id": "OzAKyoIBILrgz4V8VcpG",
      "_index": "filebeat-7.17.5-2022.08.02-000001",
      "_source": {
        "docker": {
          "container": {
            "labels": {
              "com_docker_stack_namespace": "modsecurity"
            }
          }
        },
        "ecs": {
          "version": "1.12.0"
        },
        "stream": "stderr",
        "message": "[Tue Aug 23 11:30:47.675452 2022] [:error] [pid 226:tid 139758264993536] [client 10.0.1.48:41062] [client 10.0.1.48] ModSecurity: Warning. Pattern match \"(?:^|[\\\\/])\\\\.\\\\.(?:[\\\\/]|$)\" at ARGS:test. [file \"/etc/modsecurity.d/owasp-crs/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf\"] [line \"71\"] [id \"930110\"] [msg \"Path Traversal Attack (/../)\"] [data \"Matched Data: ../ found within ARGS:test: ../\"] [severity \"CRITICAL\"] [ver \"OWASP_CRS/3.3.2\"] [tag \"modsecurity\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-lfi\"] [tag \"paranoia-level/1\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/255/153/126\"] [hostname \"modsecurity\"] [uri \"/\"] [unique_id \"YwSeR-gNfcPQLAl5gStNfAAAAQE\"]"
      }
    }
  ]
}

管道测试结果 #

{
  "docs": [
    {
      "doc": {
        "_index": "filebeat-7.17.5-2022.08.02-000001",
        "_type": "_doc",
        "_id": "OzAKyoIBILrgz4V8VcpG",
        "_source": {
          "owasp_severity": "CRITICAL",
          "owasp_message_string": "Path Traversal Attack (/../)",
          "pid": "226",
          "error": " Warning. Pattern match \"(?:^|[\\\\/])\\\\.\\\\.(?:[\\\\/]|$)\" at ARGS:test.",
          "tid": "139758264993536",
          "clientport": "41062",
          "docker": {
            "container": {
              "labels": {
                "com_docker_stack_namespace": "modsecurity"
              }
            }
          },
          "owasp_version": "OWASP_CRS/3.3.2",
          "hostname": "modsecurity",
          "ecs": {
            "version": "1.12.0"
          },
          "stream": "stderr",
          "client": "10.0.1.48",
          "client_ip": "10.0.1.48",
          "timestamp": "Tue Aug 23 11:30:47.675452 2022",
          "unique_id": "YwSeR-gNfcPQLAl5gStNfAAAAQE",
          "matching_rule_id": "930110",
          "message": "[Tue Aug 23 11:30:47.675452 2022] [:error] [pid 226:tid 139758264993536] [client 10.0.1.48:41062] [client 10.0.1.48] ModSecurity: Warning. Pattern match \"(?:^|[\\\\/])\\\\.\\\\.(?:[\\\\/]|$)\" at ARGS:test. [file \"/etc/modsecurity.d/owasp-crs/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf\"] [line \"71\"] [id \"930110\"] [msg \"Path Traversal Attack (/../)\"] [data \"Matched Data: ../ found within ARGS:test: ../\"] [severity \"CRITICAL\"] [ver \"OWASP_CRS/3.3.2\"] [tag \"modsecurity\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-lfi\"] [tag \"paranoia-level/1\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/255/153/126\"] [hostname \"waf\"] [uri \"/\"] [unique_id \"YwSeR-gNfcPQLAl5gStNfAAAAQE\"]",
          "uri": "/",
          "owasp_message_data": "Matched Data: ../ found within ARGS:test: ../",
          "tags": "[tag \"modsecurity\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-lfi\"] [tag \"paranoia-level/1\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/255/153/126\"]",
          "loglevel": "error",
          "matching_rule_file": "/etc/modsecurity.d/owasp-crs/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf",
          "matching_rule_line": "71"
        },
        "_ingest": {
          "timestamp": "2022-08-23T12:08:05.5017157Z"
        }
      }
    }
  ]
}

1. 检查FileBeat运行日志 #

直接查看FileBeat日志，确认是否存在连接失败、权限不足、管道不存在等错误：

# 容器部署场景
docker logs <filebeat-container-id>

# 主机部署场景
tail -f /var/log/filebeat/filebeat.log

重点关注含elasticsearch、pipeline、error的日志条目。

2. 验证管道条件判断逻辑 #

管道中Grok处理器的if条件为ctx?.docker.container.labels.com_docker_stack_namespace == 'modsecurity'，需确认FileBeat采集的真实日志是否包含该字段：

• 临时注释管道中的if条件，重启FileBeat后查看Discovery是否有日志出现。
• 若出现日志，说明采集的日志中该字段不存在或值不匹配，需检查FileBeat的Docker模块配置，确保容器标签被正确采集。

3. 检查Elasticsearch索引状态 #

• 确认FileBeat默认索引模板存在，避免日志写入未被Kibana识别的索引：

  curl -u xxx:xxx http://elasticsearch:9200/_template/filebeat-*
  

• 查看Elasticsearch中是否有FileBeat写入的索引：

  curl -u xxx:xxx http://elasticsearch:9200/_cat/indices/filebeat-*
  

• 若索引存在但Kibana看不到，检查Kibana的索引模式是否包含该索引。

4. 开启Elasticsearch Ingest日志排查 #

开启Ingest调试日志，查看管道执行的详细错误：

1. 修改Elasticsearch的elasticsearch.yml，添加：

   logger.org.elasticsearch.ingest: DEBUG
   

2. 重启Elasticsearch，查看与ExtractOwaspErrorFields管道相关的执行记录。

5. 确认FileBeat配置生效 #

修改filebeat.yml后，需重启FileBeat或发送重载信号确保配置生效：

# 容器部署场景
docker restart <filebeat-container-id>

# 主机部署场景
systemctl restart filebeat

内容的提问来源于stack exchange，提问作者piggeld94