如何用Boto创建EC2实例部署Docker爬虫并完成后终止实例？

Hey Marcus, great plan for running your Dockerized web crawler on EC2—your core approach is totally solid, and we can refine it to make the workflow smoother and more reliable. Let's walk through the optimized implementation, key code snippets, and answers to those nagging detail questions you likely have.

Your initial steps are on the right track, but here's a tweaked workflow to cut down on manual overhead and improve security:

• Launch an EC2 instance using an AMI pre-configured with Docker (like Amazon Linux 2 ECS-Optimized AMI, or official Ubuntu/Debian AMIs with Docker pre-installed)
• Attach an IAM Instance Role to the EC2 instance (instead of hardcoding AWS credentials) that grants permission to terminate itself
• Use a tool like Fabric or Paramiko to SSH into the instance, pull your Docker image (from ECR, Docker Hub, etc.), and start the crawler container
• Configure your crawler to automatically trigger instance termination once its task is complete (using the instance metadata service to get the local instance ID)

1. Create EC2 Instance with Boto3 #

First, use Boto3 to spin up your instance with the right AMI, security group, and IAM role:

import boto3

ec2 = boto3.resource('ec2')

# Replace these values with your own
AMI_ID = 'ami-0c55b159cbfafe1f0'  # Example: Amazon Linux 2 ECS-Optimized AMI (us-east-1)
INSTANCE_TYPE = 't2.micro'
KEY_NAME = 'your-ssh-key-pair-name'
SECURITY_GROUP_ID = 'sg-1234567890abcdef0'  # Allow SSH from your IP, no inbound ports needed for crawler if it's outbound-only
IAM_ROLE_NAME = 'CrawlerInstanceRole'  # Role with permission to terminate EC2 instances

instance = ec2.create_instances(
    ImageId=AMI_ID,
    InstanceType=INSTANCE_TYPE,
    KeyName=KEY_NAME,
    SecurityGroupIds=[SECURITY_GROUP_ID],
    IamInstanceProfile={'Name': IAM_ROLE_NAME},
    MinCount=1,
    MaxCount=1,
    UserData='''#!/bin/bash
                yum update -y
                systemctl start docker
                systemctl enable docker
             '''  # Optional: Ensure Docker is running (some AMIs might need this)
)[0]

instance.wait_until_running()
instance.reload()
print(f"Instance created: {instance.id} with public IP: {instance.public_ip_address}")

2. SSH into Instance with Fabric to Deploy Crawler #

Use Fabric v2 to connect via SSH, pull your Docker image, and start the crawler:

from fabric import Connection

# Get instance public IP from the previous step
instance_ip = instance.public_ip_address

# Connect via SSH
conn = Connection(
    host=instance_ip,
    user='ec2-user',  # Use 'ubuntu' for Ubuntu AMIs
    connect_kwargs={'key_filename': '/path/to/your-ssh-key.pem'}
)

# Pull your Docker image (replace with your image path)
conn.run('docker pull your-docker-registry/your-crawler-image:latest')

# Start the crawler container (mount necessary volumes if needed)
conn.run('docker run --rm your-docker-registry/your-crawler-image:latest')

3. Crawler Auto-Terminates Instance #

Inside your crawler code (or in a wrapper script in the Docker image), add logic to terminate the instance once the crawl is done:

Option 1: Use Boto3 inside the container (requires IAM role attached) #

import boto3
import requests

def get_instance_id():
    # Fetch instance ID from EC2 metadata service
    response = requests.get('http://169.254.169.254/latest/meta-data/instance-id')
    return response.text

def terminate_instance():
    ec2 = boto3.client('ec2')
    instance_id = get_instance_id()
    ec2.terminate_instances(InstanceIds=[instance_id])
    print(f"Terminating instance {instance_id}")

# After crawler finishes its task
if __name__ == "__main__":
    # Run crawl logic here
    print("Crawl completed successfully")
    terminate_instance()

Option 2: Use AWS CLI inside the container (simpler if you don't want to add boto3 to your image) #

Add this to your crawler's post-task script:

INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
aws ec2 terminate-instances --instance-ids $INSTANCE_ID

Here are answers to the questions you were likely about to ask:

• Do I need to hardcode AWS credentials in my crawler? No! Attach an IAM Instance Role to the EC2 instance with the ec2:TerminateInstances permission. The instance (and Docker container running on it) will automatically inherit this permission via the metadata service.
• How do I ensure my Docker image is accessible to the EC2 instance? If using Docker Hub, make sure the image is public (or configure Docker Hub credentials on the instance). For private images, use AWS ECR and grant the IAM role permission to pull from ECR.
• What if the crawler fails? Add error handling in your crawler code to still trigger instance termination, or set up an EC2 CloudWatch alarm to terminate the instance if it's idle for a certain period.
• Can I automate the entire workflow without manual intervention? Absolutely! Wrap the Boto3 instance creation, Fabric deployment, and cleanup logic into a single Python script that you can run with one command.

内容的提问来源于stack exchange，提问作者Marcus Lind