AWS定时启动EC2运行Python脚本后自动关机的实现咨询

Got it, here's a robust, cost-effective way to set up your scheduled Python workflow on EC2 (since Lambda can't handle the memory needs for parallel processing) — with full automation to start the instance, run your script, and stop it when done:

First, get your instance ready with the right setup so it can run your script and clean up after itself:

• Create an IAM Role for EC2: Attach a role to your instance with these critical permissions:
  • ec2:StopInstances: Lets the instance shut itself down once the script finishes
  • s3:GetObject (if needed): To pull your script from an S3 bucket (instead of storing it directly on the instance)
  • Add a trust policy that allows ec2.amazonaws.com to assume this role.
• Configure the Instance:
  • Pick your high-memory instance type (e.g., m5.2xlarge or whatever fits your parallel processing needs)
  • Option 1: Pre-install your Python dependencies and upload your script to the instance's EBS volume (so it's there when the instance starts)
  • Option 2: Use UserData to automate script setup on startup (great if you want to keep things dynamic):

    #!/bin/bash
    # Install required packages (adjust based on your script's needs)
    yum install -y python3-pip
    pip3 install boto3 numpy  # Replace with your dependencies
    # Pull script from S3 (update the bucket/path)
    aws s3 cp s3://your-bucket/scripts/parallel-job.py /home/ec2-user/
    # Run the script
    python3 /home/ec2-user/parallel-job.py
    # Stop the instance once the script completes (success or failure)
    aws ec2 stop-instances --instance-id $(curl -s http://169.254.169.254/latest/meta-data/instance-id)
    

  • Make sure the SSM Agent is installed (Amazon Linux 2 has it by default) — this lets you manage the instance programmatically if needed.

Use Amazon EventBridge (formerly CloudWatch Events) to kick off the workflow at your desired time:

• Create a EventBridge Rule:
  1. Go to the EventBridge console → Rules → Create rule
  2. Choose "Schedule" as the rule type, then define your schedule with a cron expression (e.g., 0 9 * * ? * runs every day at 9 AM UTC)
  3. For the target:
     • Simpler Option: Select "EC2 Instance" as the target, pick your instance, and choose "Start instance" as the action. This works if you used UserData to handle running the script and stopping the instance (like Option 2 above).
     • More Flexible Option: Use a Lambda function as the target. This lets you add logic like waiting for the instance to be fully ready before running the script, or handling errors. Here's a sample Lambda function (Python):

       import boto3
       import time
       
       ec2_client = boto3.client('ec2')
       ssm_client = boto3.client('ssm')
       TARGET_INSTANCE_ID = 'i-1234567890abcdef0'  # Replace with your instance ID
       
       def lambda_handler(event, context):
           # Start the EC2 instance
           ec2_client.start_instances(InstanceIds=[TARGET_INSTANCE_ID])
           
           # Wait until the instance is running and SSM can connect to it
           while True:
               instance_status = ec2_client.describe_instances(InstanceIds=[TARGET_INSTANCE_ID])['Reservations'][0]['Instances'][0]
               ssm_status = ssm_client.describe_instance_information(
                   InstanceInformationFilterList=[{'key': 'InstanceIds', 'valueSet': [TARGET_INSTANCE_ID]}]
               )
               
               if instance_status['State']['Name'] == 'running' and len(ssm_status['InstanceInformationList']) > 0:
                   break
               time.sleep(10)
           
           # Run your Python script via SSM Run Command
           ssm_client.send_command(
               InstanceIds=[TARGET_INSTANCE_ID],
               DocumentName='AWS-RunShellScript',
               Parameters={'commands': ['python3 /home/ec2-user/parallel-job.py']}
           )
           
           # Optional: Add logic here to wait for the script to finish, then stop the instance
           # For example, poll SSM command status until it's completed, then call ec2_client.stop_instances()
       

  • Don't forget to give your Lambda execution role permissions for ec2:StartInstances, ec2:DescribeInstances, ssm:SendCommand, and ssm:DescribeInstanceInformation.

• Manually trigger your EventBridge rule or Lambda function to test the full flow:
  1. Check if the instance starts successfully
  2. Verify your script runs (check CloudWatch Logs for the instance or SSM command output)
  3. Confirm the instance stops automatically once the script finishes
• Add error handling if needed: For example, modify your script to send an SNS alert if it fails, or adjust the stop logic to only run if the script exits successfully.

• Cost Savings: Since the instance only runs when your script is executing, you'll avoid paying for idle time. You can also consider Spot Instances for even lower costs (if your job can tolerate interruptions).
• Instance State: Stopped instances retain their EBS volume data, so you only need to install dependencies once (unless you use UserData to refresh them each time).
• Logging: Enable CloudWatch Logs for your EC2 instance or SSM commands to easily debug any issues with your script or workflow.

内容的提问来源于stack exchange，提问作者ds_user