如何通过Lambda每日自动将文件从S3迁移至EC2实例？

Alright, let's walk through exactly how to set up a daily automated file transfer from an S3 bucket to an EC2 instance using AWS Lambda. I’ve implemented this workflow a handful of times, so I’ll break it down into clear, actionable steps with code snippets and permission gotchas you don’t want to overlook.

一、Pre-Requisites #

First, make sure you have all the pieces in place:

• An S3 bucket containing the files you want to migrate
• A running EC2 instance (with SSM Agent installed and running—Amazon Linux 2 has this by default; for Ubuntu, you’ll need to install amazon-ssm-agent manually)
• An IAM role for Lambda with the right permissions (we’ll cover this next)
• The EC2 instance needs an IAM role that allows it to read from your target S3 bucket

二、Configure Lambda Permissions #

Lambda needs two core sets of permissions to pull this off. I recommend using SSM Run Command instead of direct SSH here—it’s way more secure, no need to manage SSH keys or open port 22 to the world. Here’s the IAM policy you’ll need for your Lambda execution role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::your-source-bucket",
        "arn:aws:s3:::your-source-bucket/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand",
        "ssm:GetCommandInvocation"
      ],
      "Resource": [
        "arn:aws:ec2:your-region:your-account-id:instance/your-ec2-instance-id",
        "arn:aws:ssm:your-region::document/AWS-RunShellScript"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "logs:CreateLogGroup",
      "Resource": "arn:aws:logs:your-region:your-account-id:*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:your-region:your-account-id:log-group:/aws/lambda/your-lambda-function-name:*"
    }
  ]
}

Replace all placeholders (like your-source-bucket, your-region, etc.) with your actual AWS resources.

三、Write the Lambda Function Code #

The Lambda function uses Boto3 to send a shell command to your EC2 instance via SSM, which then runs aws s3 sync to pull files from the bucket to a local directory. Here’s the Python code:

import boto3
import time

def lambda_handler(event, context):
    # Update these values to match your setup
    S3_BUCKET = "your-source-bucket"
    EC2_INSTANCE_ID = "your-ec2-instance-id"
    TARGET_DIR = "/home/ec2-user/s3-synced-files"  # Example target path on EC2
    AWS_REGION = "us-east-1"

    # Initialize SSM client
    ssm_client = boto3.client('ssm', region_name=AWS_REGION)

    # Command to sync S3 bucket to EC2 directory (--delete removes files no longer in S3)
    sync_command = f"aws s3 sync s3://{S3_BUCKET} {TARGET_DIR} --delete"

    # Send the command to EC2 via SSM
    command_response = ssm_client.send_command(
        InstanceIds=[EC2_INSTANCE_ID],
        DocumentName="AWS-RunShellScript",
        Parameters={'commands': [sync_command]}
    )

    # Wait a few seconds for the command to start, then check status
    command_id = command_response['Command']['CommandId']
    time.sleep(5)

    invocation_status = ssm_client.get_command_invocation(
        CommandId=command_id,
        InstanceId=EC2_INSTANCE_ID
    )

    # Log results and return status
    print(f"Command Status: {invocation_status['Status']}")
    if invocation_status['Status'] == 'Success':
        print(f"Sync Output: {invocation_status['StandardOutputContent']}")
        return {
            'statusCode': 200,
            'body': f"Successfully synced {S3_BUCKET} to {TARGET_DIR} on EC2 {EC2_INSTANCE_ID}"
        }
    else:
        error_msg = invocation_status['StandardErrorContent']
        print(f"Sync Failed: {error_msg}")
        return {
            'statusCode': 500,
            'body': f"Sync failed. Error: {error_msg}"
        }

Notes:

• The --delete flag in s3 sync will remove files in the EC2 directory that no longer exist in S3—remove it if you don’t want this behavior.
• Ensure the EC2 instance’s IAM role has s3:ListBucket and s3:GetObject permissions for your target bucket.

四、Set Up Daily Trigger with EventBridge #

To make this run automatically every day, use an EventBridge rule:

1. Open the AWS EventBridge console and click "Create rule"
2. Choose "Schedule" as the rule type
3. Set the schedule expression: use a cron expression like cron(0 12 * * ? *) (runs daily at UTC 12:00, adjust to your desired time zone)
4. For the target, select "Lambda function" and choose your sync function
5. Save the rule—your sync will now run automatically on schedule

五、Test & Troubleshoot #

• First, trigger the Lambda function manually via the console to verify it works. Check CloudWatch Logs for detailed output.
• Common issues to fix:
  • SSM Agent not running on EC2: Run sudo systemctl status amazon-ssm-agent to check, restart it if needed
  • EC2 lacks S3 permissions: Verify the EC2 instance’s IAM role has the right S3 policies
  • Target directory doesn’t exist: Add mkdir -p {TARGET_DIR} to the sync command to create it if missing
  • Permissions on EC2 target directory: Ensure the user running the SSM command (usually root or ec2-user) has write access

Optional Optimizations #

• Add error alerts: Use SNS to send an email/SMS when the Lambda function returns a 500 status
• Filter files: Modify the s3 sync command to include/exclude specific file types (e.g., --exclude "*" --include "*.csv")
• Quiet mode: Add --quiet to the sync command to reduce log clutter for large syncs

内容的提问来源于stack exchange，提问作者vardhan