如何基于事件字段值动态配置Logstash S3输出前缀为%{+YYYY}/%{+MM}/%{+dd}/%{+HH}格式

Got it, let's break down exactly how to set up a dynamic S3 prefix using your record_time field to generate 2017/03/09/04 as required.

Step 1: Parse the record_time field into a valid Logstash time type #

First, you need to ensure Logstash recognizes record_time as a proper timestamp (not just a string). Use the date filter to parse it—you can choose to keep the original field or overwrite Logstash's default @timestamp:

filter {
  date {
    # Match the ISO8601 format of your record_time field
    match => ["record_time", "ISO8601"]
    # Option 1: Store parsed time back to record_time (preserve original @timestamp)
    target => "record_time"
    # Option 2: Overwrite @timestamp with record_time (simpler for prefix config)
    # target => "@timestamp"
  }
}

Step 2: Configure the S3 output with dynamic prefix #

Now, set up the S3 output to use the parsed timestamp field to build your desired prefix structure.

If you chose Option 1 (kept record_time as the time field):

Specify the field explicitly in the time format syntax to generate each segment of the prefix:

output {
  s3 {
    access_key_id => "your_access_key_here"
    secret_access_key => "your_secret_key_here"
    bucket => "your_target_bucket"
    # Use the record_time field to generate each part of the path
    prefix => "%{+YYYY:record_time}/%{+MM:record_time}/%{+dd:record_time}/%{+HH:record_time}"
    # Or use a single formatted string for brevity:
    # prefix => "%{+YYYY/MM/dd/HH:record_time}"
    codec => "json_lines" # Adjust codec based on your data format needs
  }
}

If you chose Option 2 (overwrote @timestamp):

Since Logstash uses @timestamp by default for time-based formatting, you can simplify the prefix config:

output {
  s3 {
    access_key_id => "your_access_key_here"
    secret_access_key => "your_secret_key_here"
    bucket => "your_target_bucket"
    prefix => "%{+YYYY}/%{+MM}/%{+dd}/%{+HH}"
    codec => "json_lines"
  }
}

How it works #

• The date filter converts the string value of record_time into a Logstash-native timestamp object, which enables time-based formatting.
• The %{+FORMAT:FIELD} syntax tells Logstash to format the specified timestamp field (record_time or @timestamp) using the given date pattern, building the dynamic S3 path exactly as you need it.

Testing this with your sample event {"record_time":"2017-03-09T04:07:51.520Z"} will generate the prefix 2017/03/09/04 perfectly.

内容的提问来源于stack exchange，提问作者Forece85