ELK FileBeat配置异常：Pipeline未生效问题修复

在同一filebeat.yml中配置了多个AWS CloudWatch输入及对应索引，Pipeline测试无报错且能成功解析文档ID，但Pipeline并未执行，Kibana中无法看到处理后的字段。

原配置 #

#=========================== Filebeat inputs =============================
filebeat.inputs:
- type: aws-cloudwatch
  enabled: true
  access_key_id: "xxxxxxxxxxxxx"
  secret_access_key: "xxxxxxxxxxxx"
  region_name: "xxxxxxxxx"
  log_group_arn: "arn:aws:logs:xxxxxxxx:apache_logs"
  log_group_name: "xxxxxxxxx/apache"
  fields:
    name: "prod-cloudwatch_apache_logs"
  ignore_older: 720h
  start_position: end
- type: aws-cloudwatch
  enabled: true
  access_key_id: "xxxxxxxxxxxxx"
  secret_access_key: "xxxxxxxxxxxx"
  region_name: "xxxxxxxxx"
  log_group_arn: "arn:aws:logs:xxxxxxxx:apache_logs"
  log_group_name: "xxxxxxxxx/apache"
  fields:
    name: "uat-cloudwatch_apache_logs"
  ignore_older: 720h
  start_position: end

output.elasticsearch:
  hosts: ["https://xxxxxxx.aws.elastic-cloud.com:xxxxxx"]
  protocol: "https"
  username: "${ESUSER}"
  password: "${ESPASS}"
  output.elasticsearch.allow_older_versions: true
  indices:
   - index: "cloudwatch_uat_apache_logs-%{+yyyy.MM}"
     when.contains:
      fields.name: "uat-cloudwatch_apache_logs"
   - index: "cloudwatch_prod_apache_logs%{+yyyy.MM}"
     when.contains:
      fields.name: "prod-cloudwatch_apache_logs"
  pipelines:
    - pipeline: "apache_pipeline"
      when:
       has_fields: ['name.prod-cloudwatch_apache_logs']

配置错误分析与修复方案 #

• Pipeline匹配条件错误：原条件写法无效，需通过fields.name的值匹配，同时覆盖UAT环境；
• 参数缩进错误：allow_older_versions应与hosts等同级，无需重复嵌套output.elasticsearch前缀；
• 索引格式不一致：Prod索引名缺少连接符-，需统一格式。

修复后的配置 #

#=========================== Filebeat inputs =============================
filebeat.inputs:
- type: aws-cloudwatch
  enabled: true
  access_key_id: "xxxxxxxxxxxxx"
  secret_access_key: "xxxxxxxxxxxx"
  region_name: "xxxxxxxxx"
  log_group_arn: "arn:aws:logs:xxxxxxxx:apache_logs"
  log_group_name: "xxxxxxxxx/apache"
  fields:
    name: "prod-cloudwatch_apache_logs"
  ignore_older: 720h
  start_position: end
- type: aws-cloudwatch
  enabled: true
  access_key_id: "xxxxxxxxxxxxx"
  secret_access_key: "xxxxxxxxxxxx"
  region_name: "xxxxxxxxx"
  log_group_arn: "arn:aws:logs:xxxxxxxx:apache_logs"
  log_group_name: "xxxxxxxxx/apache"
  fields:
    name: "uat-cloudwatch_apache_logs"
  ignore_older: 720h
  start_position: end

output.elasticsearch:
  hosts: ["https://xxxxxxx.aws.elastic-cloud.com:xxxxxx"]
  protocol: "https"
  username: "${ESUSER}"
  password: "${ESPASS}"
  allow_older_versions: true
  indices:
   - index: "cloudwatch_uat_apache_logs-%{+yyyy.MM}"
     when.contains:
      fields.name: "uat-cloudwatch_apache_logs"
   - index: "cloudwatch_prod_apache_logs-%{+yyyy.MM}"
     when.contains:
      fields.name: "prod-cloudwatch_apache_logs"
  pipelines:
    - pipeline: "apache_pipeline"
      when.contains:
        fields.name: ["prod-cloudwatch_apache_logs", "uat-cloudwatch_apache_logs"]

内容的提问来源于stack exchange，提问作者user1428716