如何为AWS Batch作业动态配置适配S3 Glacier深度归档文件大小的临时存储

Great question! This is a common scenario when dealing with large, variable-sized archives in Glacier, and AWS Batch + Lambda gives you a flexible way to handle dynamic storage needs. Let’s walk through the step-by-step solution:

Step 1: Verify Archive Restoration & Get File Size #

First, your Lambda function needs to confirm the Glacier archive has finished restoring before proceeding (since you want to trigger unpacking immediately after recovery). Use the S3 API to check the restore status and fetch the exact file size without downloading the entire archive:

import boto3

s3_client = boto3.client('s3')

def get_archive_details(bucket, key):
    attrs = s3_client.get_object_attributes(
        Bucket=bucket,
        Key=key,
        Attributes=['ObjectSize', 'RestoreStatus']
    )
    # Check if restore is complete (expiry date exists when restoration is done)
    is_restored = bool(attrs['RestoreStatus'].get('RestoreExpiryDate'))
    file_size_gb = attrs['ObjectSize'] / (1024 ** 3)
    return is_restored, round(file_size_gb, 2)

Step 2: Calculate Required EBS Storage #

Tar.gz files typically expand 2-3x when unpacked (depending on compression rate). To avoid running out of space, calculate an EBS volume size with enough redundancy. For example:

• Use 2x the archive size as a baseline
• Set a minimum threshold (e.g., 10GB) to handle small archives efficiently

import math

def calculate_ebs_size(archive_size_gb):
    # Adjust multiplier based on your archive's compression behavior
    multiplier = 2
    ebs_size = math.ceil(archive_size_gb * multiplier)
    # Ensure minimum size for small files
    return max(ebs_size, 10)

Step 3: Dynamically Configure EBS Storage in AWS Batch #

The key here is you don’t need to mess with static Launch Templates. AWS Batch lets you override volume configurations at job submission time via containerOverrides, which is perfect for your dynamic storage needs.

Full Lambda Example Code #

import boto3
import math

s3_client = boto3.client('s3')
batch_client = boto3.client('batch')

def lambda_handler(event, context):
    # Extract S3 object details from the trigger event (e.g., S3 Restore Complete event)
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = event['Records'][0]['s3']['object']['key']
    
    # Check restoration status and get file size
    is_restored, archive_size_gb = get_archive_details(bucket, key)
    if not is_restored:
        return {"status": "restoration_pending", "archive_key": key}
    
    # Calculate required EBS size
    ebs_size_gb = calculate_ebs_size(archive_size_gb)
    
    # Submit Batch job with dynamic EBS volume
    submit_response = batch_client.submit_job(
        jobName=f"unpack-{key.split('/')[-1]}",
        jobQueue="your-job-queue-name",
        jobDefinition="your-unpack-job-definition",
        containerOverrides={
            'environment': [
                {'name': 'SOURCE_BUCKET', 'value': bucket},
                {'name': 'SOURCE_KEY', 'value': key},
                {'name': 'TARGET_BUCKET', 'value': 'your-target-s3-bucket'}
            ],
            'volumes': [
                {
                    'name': 'unpack-storage',
                    'ebsVolumeConfiguration': {
                        'volumeSize': ebs_size_gb,
                        'volumeType': 'gp3',  # Balances cost and performance
                        'deleteOnTermination': True  # Clean up after job finishes
                    }
                }
            ],
            'mountPoints': [
                {
                    'sourceVolume': 'unpack-storage',
                    'containerPath': '/mnt/unpack'
                }
            ]
        }
    )
    
    return {
        "status": "job_submitted",
        "job_id": submit_response['jobId'],
        "ebs_size_gb": ebs_size_gb,
        "archive_size_gb": archive_size_gb
    }

def get_archive_details(bucket, key):
    attrs = s3_client.get_object_attributes(
        Bucket=bucket,
        Key=key,
        Attributes=['ObjectSize', 'RestoreStatus']
    )
    is_restored = bool(attrs['RestoreStatus'].get('RestoreExpiryDate'))
    file_size_gb = attrs['ObjectSize'] / (1024 ** 3)
    return is_restored, round(file_size_gb, 2)

def calculate_ebs_size(archive_size_gb):
    multiplier = 2
    ebs_size = math.ceil(archive_size_gb * multiplier)
    return max(ebs_size, 10)

Batch Job Definition Setup #

Create an EC2-type Job Definition (Fargate has fixed storage limits) with a base configuration—Lambda will override the EBS size at submission time:

{
  "jobDefinitionName": "unpack-tar-archive",
  "type": "container",
  "platformCapabilities": ["EC2"],
  "containerProperties": {
    "image": "your-custom-docker-image",
    "vcpus": 4,
    "memory": 8000,
    "mountPoints": [
      {
        "sourceVolume": "unpack-storage",
        "containerPath": "/mnt/unpack"
      }
    ],
    "volumes": [
      {
        "name": "unpack-storage",
        "ebsVolumeConfiguration": {
          "volumeSize": 10,  # Base size (will be overridden)
          "volumeType": "gp3",
          "deleteOnTermination": true
        }
      }
    ],
    "environment": [
      {"name": "SOURCE_BUCKET", "value": ""},
      {"name": "SOURCE_KEY", "value": ""},
      {"name": "TARGET_BUCKET", "value": ""}
    ]
  }
}

Custom Docker Image #

Build an image with tools to download, unpack, and sync files to S3:

FROM amazonlinux:2

# Install required utilities
RUN yum update -y && yum install -y tar aws-cli && yum clean all

# Job command: download archive, unpack, sync to target bucket
CMD ["bash", "-c", "\
    aws s3 cp s3://$SOURCE_BUCKET/$SOURCE_KEY /mnt/unpack/archive.tar.gz && \
    mkdir -p /mnt/unpack/extracted && \
    tar xzf /mnt/unpack/archive.tar.gz -C /mnt/unpack/extracted && \
    aws s3 sync /mnt/unpack/extracted/ s3://$TARGET_BUCKET/extracted/$(basename $SOURCE_KEY .tar.gz)/ && \
    rm -rf /mnt/unpack/* \
"]

Key Considerations #

• Instance Compatibility: Use EC2 instances in your Batch Compute Environment that support attaching EBS volumes (most modern types like t3, m5, c5 work perfectly).
• Storage Multiplier: Adjust the 2x multiplier based on your actual archive compression rates—if your files are highly compressed, you may need 3x or more.
• Cost Control: The deleteOnTermination: true flag ensures EBS volumes are deleted after the job finishes, preventing unused storage costs.
• Permissions: Grant Lambda permissions for batch:SubmitJob and s3:GetObjectAttributes, and ensure Batch EC2 instances have s3:GetObject and s3:PutObject access.

内容的提问来源于stack exchange，提问作者Quentin Chartreux