Logstash配置从S3获取ALB日志并按目标组拆分索引时出现索引格式错误问题求助

API 探路者

2026-4-20

我最近在配置Logstash从S3桶拉取Application Load Balancer（ALB）的日志，我的ALB关联了多个目标组，需求是给每个目标组的日志创建独立的Elasticsearch索引。

我编写了Logstash配置，在filter阶段解析ALB日志内容，通过匹配target_group_arn字段的值，给对应目标组的日志添加target_group标识字段；接着在output阶段根据这个字段判断，将日志写入不同的索引。但配置生效后，Logstash抛出了索引格式错误的警告，索引也没有成功创建。如果去掉这些过滤和索引拆分的逻辑，我能正常从S3获取到ALB日志，所以问题应该出在我的过滤规则或索引配置上。

以下是我的Logstash配置：

input {
  s3 {
    access_key_id => "credentials"
    secret_access_key => "credentials"
    bucket => "bucket_name"
    region => "region_name"
    prefix => "ALB-logs/AWSLogs/5298/elasticloadbalancing/region_name/"
  }
}

filter {
  # Parse log lines with a grok filter
  grok {
    match => {
      "message" => '%{DATA:timestamp} %{WORD:elb_name}/%{DATA} %{IPORHOST:client_ip}:%{NUMBER:client_port} %{IPORHOST:backend_ip}:%{NUMBER:backend_port} %{NUMBER:request_processing_time} %{NUMBER:backend_processing_time} %{NUMBER:response_processing_time} %{NUMBER:elb_status_code} %{NUMBER:backend_status_code} %{NUMBER:received_bytes} %{NUMBER:sent_bytes} "%{WORD:http_method} %{DATA:request_uri} HTTP/%{NUMBER:http_version}" "%{DATA:user_agent}" %{DATA:ssl_cipher} %{DATA:ssl_protocol} %{DATA:target_group_arn} "%{DATA:trace_id}" "%{DATA:host}" "%{DATA:ssl_certificate_arn}" %{NUMBER:ssl_cipher_bits} %{DATA:timestamp} "%{DATA:action}" "%{DATA:waf_response_code}" "%{DATA:waf_message}" "%{DATA:backend_description}" "%{DATA:elb_response_code}" "%{DATA:elb_response_description}" "%{DATA:elb_target_ip}" "%{DATA:elb_target_port}"'
    }
  }

  # Add a field to indicate the target group
  if [target_group_arn] =~ /app1/ {
    mutate {
      add_field => { "target_group" => "app1" }
    }
  } else if [target_group_arn] =~ /frontend/ {
    mutate {
      add_field => { "target_group" => "frontend" }
    }
  } else if [target_group_arn] =~ /Backend/ {
    mutate {
      add_field => { "target_group" => "Backend" }
    }
  } else if [target_group_arn] =~ /ORM/ {
    mutate {
      add_field => { "target_group" => "ORM" }
    }
  } else if [target_group_arn] =~ /OASC/ {
    mutate {
      add_field => { "target_group" => "OASC" }
    }
  } else if [target_group_arn] =~ /security/ {
    mutate {
      add_field => { "target_group" => "security" }
    }
  } else if [target_group_arn] =~ /TOB/ {
    mutate {
      add_field => { "target_group" => "TOB" }
    }
  }

  # Add more conditions for other target groups as needed...
}

output {
  # Output to Elasticsearch with separate indices based on target group
  if ([target_group] in ["app1"]) {
    elasticsearch {
      hosts => ["localhost:9200"]
      index => "alb-app-%{+YYYY.MM.dd}"
      # Additional configuration for Elasticsearch output
    }
  } else if ([target_group] in ["frontend", "Backend"]) {
    elasticsearch {
      hosts => ["localhost:9200"]
      index => "alb-prod-%{+YYYY.MM.dd}"
      # Additional configuration for Elasticsearch output
    }
  } else if ([target_group] in ["ORM", "OASC"]) {
    elasticsearch {
      hosts => ["localhost:9200"]
      index => "alb-orm-%{+YYYY.MM.dd}"
      # Additional configuration for Elasticsearch output
    }
  } else if ([target_group] in ["security"]) {
    elasticsearch {
      hosts => ["localhost:9200"]
      index => "alb-security-%{+YYYY.MM.dd}"
      # Additional configuration for Elasticsearch output
    }
  }

  # Add more conditions for other target groups as needed...
  else {
    elasticsearch {
      hosts => ["localhost:9200"]
      index => "alb-default-%{+yyyy.MM.dd}"
      manage_template => false
    }
  }

  stdout { codec => rubydebug }
}

Logstash抛出的警告信息如下：

[WARN ] 2024-02-29 18:35:06.186 [[main]>worker0] elasticsearch - Badly formatted index, after interpolation still contains placeholder: [%{[@metadata][beat]}-2024.02.29]; event: `{"@timestamp"=>2024-02-29T13:01:18.463720661Z, "@metadata"=>{"s3"=>{"key"=>"ALB-logs/AWSLogs/5298/elasticloadbalancing/region_name/2024/02/03/5298_elasticloadbalancing_region_name_app.Application-LB.6729648b993f37bb_20240203T0430Z_13.126.185.122_4xljf1i6.log.gz"}}, "@version"=>"1", "message"=>"https 2024-02-03T04:25:53.992906Z app/Application-LB/6729648b993f37bb 115.97.253.187:43816 192.168.11.215:8080 0.001 0.001 0.000 404 404 622 1921 \"GET https://demo.example.net:443/mytag_js.js HTTP/1.1\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36\" TLS_AES_128_GCM_SHA256 TLSv1.3 arn:aws:elasticloadbalancing:region_name:5298:targetgroup/app1/50e7f17a37ab3cdd \"Root=1-65bdc051-72c953f82263264c\" \"demo.example.net\" \"arn:aws:acm:region_name:5298:certificate/14eda-6242-43dc-e-a5fc75a55b2f\" 0 2024-02-03T04:25:53.990000Z \"waf,forward\" \"-\" \"-\" \"192.168.11.215:8080\" \"404\" \"-\" \"-\"", "event"=>{"original"=>"https 2024-02-03T04:25:53.992906Z app/Application-LB/67296993f37bb 115.97.253.187:43816 192.168.11.215:8080 0.001 0.001 0.000 404 404 622 1921 \"GET https://demo.example.net:443/mytag_js.js HTTP/1.1\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36\" TLS_AES_128_GCM_SHA256 TLSv1.3 arn:aws:elasticloadbalancing:region_name:5298:targetgroup/app1/50e7f17a37ab3cdd \"Root=1-65bdc051-72c953f82ad59fbd4263264c\" \"demo.example.net\" \"arn:aws:acm:region_name:5298:certificate/14ed3a7a-6242-4c-9d5e-a55a55b2f\" 0 2024-02-03T04:25:53.990000Z \"waf,forward\" \"-\" \"-\" \"192.168.11.215:8080\" \"404\" \"-\" \"-\""}, "tags"=>["_grokparsefailure"]}`

麻烦各位帮我排查一下问题所在，或者纠正我的配置错误，谢谢大家！

备注：内容来源于stack exchange，提问作者HaridharanS