Python多进程子进程挂起问题求助（涉及BOTO3、PARAMIKO）

Hey there, let's work through this unresponsive process pool issue together—those stuck worker processes are such a frustrating roadblock when you’ve got a clear workflow mapped out!

First, let’s break down the most common culprits for this kind of hang, along with actionable fixes:

1. Shared AWS SDK Resources Causing Conflicts #

A super common pitfall is reusing a parent-process boto3 client/session across child processes. AWS SDKs aren’t always thread/process-safe, and sharing connections can lead to deadlocks or unresponsive workers.

Fix: Initialize a fresh boto3 session inside each worker function, not in the parent process. This ensures each worker has its own isolated connection to AWS.

2. Missing Error Handling & Cleanup #

If a worker hits an uncaught exception mid-process (e.g., EC2 instance launch fails, script execution hangs), it might freeze instead of exiting. Without cleanup logic, stuck instances can also tie up AWS resources and block subsequent tasks.

Fix: Wrap your entire worker workflow in a try/except block, and add logic to terminate EC2 instances even if something goes wrong.

3. Unreliable Script Completion Detection #

If your worker doesn’t have a rock-solid way to confirm the AMI script finished, it might wait indefinitely. Relying on instance status alone isn’t enough—you need explicit confirmation the script ran to completion.

Fix: Use one of these methods to verify script success:

• Have the script add a custom tag (e.g., ScriptCompleted: Yes) to the instance when done, then poll for that tag in your worker.
• Use AWS SSM Run Command to execute the script (instead of relying on AMI startup scripts) and wait for the command execution to finish via SSM waiters.
• Have the script write a completion marker to an S3 object, then poll that object in your worker.

4. Overloaded Process Pool or AWS Quotas #

If you set your process pool size too high, you might hit local system limits (CPU/memory) or AWS EC2 instance launch quotas. Either scenario can cause workers to hang while waiting for resources.

Fix:

• Check your AWS account’s EC2 running instance quota for your region (you can find this in the AWS Console’s Service Quotas section). Set your pool size to stay below this limit.
• Match the pool size to your local machine’s available CPU cores (e.g., processes=4 for a 4-core machine) to avoid overwhelming your system.

Example Fixed Worker Script #

Here’s a revised version of your workflow incorporating all these fixes:

import boto3
import time
from multiprocessing import Pool

def process_target(target):
    # Initialize fresh AWS clients for this worker
    ec2 = boto3.client('ec2')
    instance_id = None

    try:
        # 1. Launch EC2 instance
        launch_response = ec2.run_instances(
            ImageId='your-ami-id-here',
            InstanceType='t2.micro',
            MinCount=1,
            MaxCount=1,
            # Optional: Explicitly trigger your script via UserData if needed
            UserData='#!/bin/bash\n/path/to/your/ami-script.sh'
        )
        instance_id = launch_response['Instances'][0]['InstanceId']
        print(f"Launched instance {instance_id} for target: {target}")

        # Wait for instance to be running and healthy
        ec2.get_waiter('instance_running').wait(InstanceIds=[instance_id])
        ec2.get_waiter('instance_status_ok').wait(InstanceIds=[instance_id])

        # 2. Wait for script completion (using tag-based confirmation)
        print(f"Waiting for script to finish on {instance_id}...")
        while True:
            instance_details = ec2.describe_instances(InstanceIds=[instance_id])['Reservations'][0]['Instances'][0]
            tags = {tag['Key']: tag['Value'] for tag in instance_details.get('Tags', [])}
            if tags.get('ScriptCompleted') == 'Yes':
                break
            time.sleep(30)

        # 3. Terminate the instance
        ec2.terminate_instances(InstanceIds=[instance_id])
        print(f"Successfully processed {target} - terminated {instance_id}")
        return f"Success: {target}"
    
    except Exception as e:
        # Cleanup: Terminate instance if it exists, even on failure
        if instance_id:
            try:
                ec2.terminate_instances(InstanceIds=[instance_id])
                print(f"Terminated stuck instance {instance_id} after error: {str(e)}")
            except:
                print(f"Failed to terminate {instance_id} after error: {str(e)}")
        print(f"Failed to process {target}: {str(e)}")
        return f"Failed: {target} - {str(e)}"

if __name__ == '__main__':
    target_cos = ["target1", "target2", "target3", "target4"]
    # Set pool size based on AWS quotas and local resources
    with Pool(processes=4) as pool:
        results = pool.map(process_target, target_cos)
    
    # Print final results summary
    for result in results:
        print(result)

Bonus Tips #

• Add detailed logging (instead of print statements) using Python’s logging module—this will help you pinpoint exactly where a worker is getting stuck.
• Test with a small target_cos list first to validate the workflow before scaling up.

内容的提问来源于stack exchange，提问作者Travis Millburn