通过EventBridge传递S3触发元数据至ECS Fargate任务遇空值问题

尝试通过EventBridge将触发事件的S3存储桶名称和对象键传递给ECS Fargate任务时，获取到的环境变量为空字符串，导致Python脚本调用S3接口时参数验证失败，最终通过修改InputPathsMap中的路径解决问题。

错误配置代码 #

aws events put-targets --rule s3-to-rds --targets '{"Id": "", "Arn": "ecs cluster ARN", "RoleArn": "eventbridge role ARN with permissions", "EcsParameters": {"TaskDefinitionArn": "task ARN", "TaskCount": 1, "LaunchType": "FARGATE", "NetworkConfiguration": {"awsvpcConfiguration": {"Subnets": [""], "AssignPublicIp": "DISABLED"}}, "Group": "default"}, "InputTransformer": {"InputPathsMap": {"s3-bucket": "$.detail.requestParameters.bucketName", "s3-key": "$.detail.requestParameters.key"}, "InputTemplate": "{\"containerOverrides\": [{\"name\": \"containername\", \"environment\": [{\"name\": \"S3_BUCKET_NAME\", \"value\": \"<s3-bucket>\"}, {\"name\": \"S3_OBJECT_KEY\", \"value\": \"<s3-key>\"}]}]}"}}'

Python代码片段 #

import os
import logging
import boto3

S3_BUCKET_NAME = os.environ.get('S3_BUCKET_NAME')
logging.info(f"S3_BUCKET_NAME: {S3_BUCKET_NAME}")
S3_OBJECT_KEY = os.environ.get('S3_OBJECT_KEY')
logging.info(f"S3_OBJECT_KEY: {S3_OBJECT_KEY}")

s3 = boto3.client('s3')
response = s3.get_object(Bucket=S3_BUCKET_NAME, Key=S3_OBJECT_KEY)

（注：原代码中第二个日志打印的变量名存在错误，已修正为S3_OBJECT_KEY）

CloudTrail报错日志 #

August 30, 2023 at 17:03 (UTC+2:00) response = handler(**kwargs)    tess
...
botocore.exceptions.ParamValidationError: Parameter validation failed:  tess
Invalid bucket name "<$s3-bucket>": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:(s3|s3-object-lambda):[a-z\-0-9]*:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-.]{1,63}$|^arn:(aws).*:s3-outposts:[a-z\-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9\-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9\-]{1,63}$"    tess
...
2023-08-30 15:03:52,840 - INFO - S3_BUCKET_NAME:    tess
2023-08-30 15:03:52,840 - INFO - S3_BUCKET_NAME:    tess

解决方法 #

将InputPathsMap中的路径修改为S3事件正确的JSON结构路径：

• 原路径："s3-bucket": "$.detail.requestParameters.bucketName" → 修改为 "s3-bucket": "$.detail.bucket.name"
• 原路径："s3-key": "$.detail.requestParameters.key" → 修改为 "s3-key": "$.detail.object.key"

修正后的配置代码：

aws events put-targets --rule s3-to-rds --targets '{"Id": "", "Arn": "ecs cluster ARN", "RoleArn": "eventbridge role ARN with permissions", "EcsParameters": {"TaskDefinitionArn": "task ARN", "TaskCount": 1, "LaunchType": "FARGATE", "NetworkConfiguration": {"awsvpcConfiguration": {"Subnets": [""], "AssignPublicIp": "DISABLED"}}, "Group": "default"}, "InputTransformer": {"InputPathsMap": {"s3-bucket": "$.detail.bucket.name", "s3-key": "$.detail.object.key"}, "InputTemplate": "{\"containerOverrides\": [{\"name\": \"containername\", \"environment\": [{\"name\": \"S3_BUCKET_NAME\", \"value\": \"<s3-bucket>\"}, {\"name\": \"S3_OBJECT_KEY\", \"value\": \"<s3-key>\"}]}]}"}}'

内容的提问来源于stack exchange，提问作者Vandalism