Filebeat配置：如何为不同日志路径指定独立Elasticsearch索引？

我正在收集两个路径下的日志：

• /var/log/containers/*.log
• /var/log/agents/*.log

希望将每个路径的日志发送到Elasticsearch的不同索引，但尝试了多种配置都未生效：

尝试的配置1 #

output.elasticsearch:
  protocol: http
  hosts: ["elasticsearch:9200"]
  compression_level: 1
  indices:
  - index: "agent-logs"
    when:
     contains:
      log.file.path: "/var/log/agents/*.log"
  - index: "container-logs"
    when:
     contains:
      log.file.path: "/var/log/containers/*.log"

尝试的配置2 #

output.elasticsearch:
  protocol: http
  hosts: ["elasticsearch:9200"]
  compression_level: 1
  indices:
  - index: "agent-logs"
    when.contains:
      log.file.path: "/var/log/agents/*.log"
  - index: "container-logs"
    when.contains:
      log.file.path: "/var/log/containers/*.log"

尝试的配置3 #

output.elasticsearch:
  protocol: http
  hosts: ["elasticsearch:9200"]
  compression_level: 1
  indices:
  - index: "agent-logs"
    when.equals:
      log.file.path: "/var/log/agents/*.log"
  - index: "container-logs"
    when.equals:
      log.file.path: "/var/log/containers/*.log"

问题原因 #

配置无效的核心原因是：log.file.path字段存储的是具体的日志文件路径（比如/var/log/agents/agent_01.log），而非你在input中配置的带通配符的路径。用contains匹配带*.log的字符串、或用equals完全匹配带通配符的路径，都无法匹配到真实的文件路径。

解决方案 #

方案1：用contains匹配路径前缀

直接匹配目录前缀，无需通配符：

output.elasticsearch:
  protocol: http
  hosts: ["elasticsearch:9200"]
  compression_level: 1
  indices:
  - index: "agent-logs"
    when.contains:
      log.file.path: "/var/log/agents/"
  - index: "container-logs"
    when.contains:
      log.file.path: "/var/log/containers/"

方案2：用regexp匹配正则表达式

如果需要精确匹配.log后缀，可使用正则：

output.elasticsearch:
  protocol: http
  hosts: ["elasticsearch:9200"]
  compression_level: 1
  indices:
  - index: "agent-logs"
    when.regexp:
      log.file.path: "^/var/log/agents/.+\\.log$"
  - index: "container-logs"
    when.regexp:
      log.file.path: "^/var/log/containers/.+\\.log$"

方案3：在input阶段添加自定义字段（推荐）

在Filebeat的input配置中，给不同路径的日志添加自定义字段，后续output根据该字段判断索引，维护性更强：

filebeat.inputs:
- type: filestream
  id: container-logs-input
  paths:
    - "/var/log/containers/*.log"
  fields:
    log_source: "container"
  fields_under_root: true

- type: filestream
  id: agent-logs-input
  paths:
    - "/var/log/agents/*.log"
  fields:
    log_source: "agent"
  fields_under_root: true

output.elasticsearch:
  protocol: http
  hosts: ["elasticsearch:9200"]
  compression_level: 1
  indices:
  - index: "agent-logs"
    when.equals:
      log_source: "agent"
  - index: "container-logs"
    when.equals:
      log_source: "container"

内容的提问来源于stack exchange，提问作者user15937765