如何为Amazon Kinesis Data Firehose设置S3目标自定义文件名格式？

First off, you're right—there's no direct setting in Kinesis Data Firehose to fully customize the S3 object filename format. As you found in the FAQs, Firehose defaults to a pattern like DeliveryStreamName-DeliveryStreamVersion-YYYY-MM-DD-HH-MM-SS-RandomString for its output files. But don't worry, there are two reliable workarounds to get the filename format you need:

Option 1: Post-Process with Lambda (Recommended) #

This approach keeps Firehose's built-in batching, error handling, and scaling capabilities, while letting you rename files after they land in S3. Here's how to set it up:

• Step 1: Configure S3 Event Notifications
  Go to your target S3 bucket, add an event trigger that fires when new objects are created (filter by the prefix/suffix used by your Firehose delivery stream to avoid unnecessary triggers). Set the target to a new Lambda function.

• Step 2: Build the Lambda Rename Logic
  Your function will need to:

  1. Fetch the newly created Firehose object using the S3 event details.
  2. Generate your custom filename (e.g., include XML metadata, timestamps, or business identifiers from your processed data).
  3. Copy the object content to the same bucket using the new filename.
  4. Delete the original Firehose-generated object.

  Example snippet (Python):

  import boto3
  
  s3 = boto3.client('s3')
  
  def lambda_handler(event, context):
      # Get original object details
      record = event['Records'][0]
      bucket = record['s3']['bucket']['name']
      old_key = record['s3']['object']['key']
  
      # Generate custom filename (adjust this to your needs)
      custom_filename = f"processed_{old_key.split('-')[-1]}.json" # Example pattern
  
      # Copy and delete
      s3.copy_object(CopySource={'Bucket': bucket, 'Key': old_key}, Bucket=bucket, Key=custom_filename)
      s3.delete_object(Bucket=bucket, Key=old_key)
  

• Key Considerations

  • Grant your Lambda function permissions for s3:GetObject, s3:PutObject, and s3:DeleteObject on the target bucket.
  • For large files, use S3 multipart uploads to avoid Lambda memory/timeouts.
  • Add idempotency checks (e.g., check if the custom file already exists) to handle duplicate event triggers.

Option 2: Route Firehose Data Through Lambda (Full Control) #

If you need end-to-end control over the filename before writing to S3, you can configure Firehose to send data to a Lambda function first, instead of directly to S3. The Lambda will handle processing and write directly to S3 with your custom filename:

• Step 1: Reconfigure Firehose Target
  Update your Firehose delivery stream to use a Lambda function as its target, instead of S3.

• Step 2: Process and Write to S3 in Lambda
  Your function will receive batched records from Firehose (note: data is Base64-encoded), process them, and write to S3 with your desired filename. This lets you group records into files by custom criteria (e.g., per XML ID, per hour, etc.).

• Key Considerations

  • You'll need to handle batching, retries, and error handling yourself (since you're bypassing Firehose's S3 batching logic).
  • Ensure your Lambda has enough memory and timeout to handle large batches.
  • Use Firehose's dead-letter queue to catch failed records and avoid data loss.

Partial Customization (If Full Filename Control Isn't Critical) #

If you just need to add a custom prefix (not full filename control), Firehose lets you configure dynamic prefixes using variables like !{timestamp:yyyy} or !{partitionKey}. For example, setting the S3 prefix to processed/xml/!{partitionKey}/!{timestamp:yyyy-MM-dd}/ will organize files by your custom partition key (e.g., XML document ID) and date, while keeping Firehose's default filename format for the rest.

内容的提问来源于stack exchange，提问作者mccarthyj