如何通过Boto3获取云盘剩余空间占比并实现自动扩容？

Your initial code is a solid starting point, but it’s missing two critical pieces: checking filesystem-level free space (the EC2 API doesn’t expose this directly) and resizing the filesystem after expanding the volume (AWS volume resizing doesn’t automatically adjust the underlying filesystem). Here’s a complete, functional script that addresses these gaps, plus error handling and best practices.

Prerequisites First #

Before running this script, ensure your EC2 instance meets these requirements:

• SSM Agent Installed: Default on Amazon Linux 2, Ubuntu, RHEL, and most modern AMIs. If missing, install it manually via AWS docs.
• IAM Permissions: Attach an IAM role to the instance with these policies (or a custom policy with equivalent permissions):
  • AmazonSSMManagedInstanceCore (for running commands via SSM)
  • Permissions to modify EC2 volumes (ec2:ModifyVolume, ec2:DescribeVolumes)

Complete Script #

import boto3
from botocore.exceptions import ClientError

def get_disk_usage(instance_id, region):
    """Retrieve disk usage stats from the EC2 instance using SSM Run Command"""
    ssm_client = boto3.client('ssm', region_name=region)
    try:
        # Run df -P to get POSIX-formatted disk usage data
        response = ssm_client.send_command(
            InstanceIds=[instance_id],
            DocumentName='AWS-RunShellScript',
            Parameters={'commands': ["df -P | grep -v Filesystem | awk '{print $1, $5, $6}'"]}
        )
        command_id = response['Command']['CommandId']
        
        # Wait for command execution to finish
        invocation = ssm_client.get_command_invocation(
            CommandId=command_id,
            InstanceId=instance_id
        )
        
        # Parse output into a dictionary: {device: {usage: int, mount: str}}
        disk_data = {}
        for line in invocation['StandardOutputContent'].strip().split('\n'):
            device, usage, mount_point = line.split()
            disk_data[device] = {
                'usage_percent': int(usage.rstrip('%')),
                'mount_point': mount_point
            }
        return disk_data
    except ClientError as e:
        print(f"Error fetching disk usage: {e.response['Error']['Message']}")
        return None

def get_volume_device_mapping(instance):
    """Map EC2 volume IDs to their attached device names on the instance"""
    volume_map = {}
    for volume in instance.volumes.all():
        for attachment in volume.attachments:
            volume_map[volume.id] = attachment['Device']
    return volume_map

def resize_filesystem(instance_id, region, device, mount_point):
    """Resize the filesystem on the instance after volume expansion"""
    ssm_client = boto3.client('ssm', region_name=region)
    try:
        # Detect filesystem type
        fs_type_response = ssm_client.send_command(
            InstanceIds=[instance_id],
            DocumentName='AWS-RunShellScript',
            Parameters={'commands': [f"df -T {device} | grep -v Type | awk '{{print $2}}'"]}
        )
        fs_type_invocation = ssm_client.get_command_invocation(
            CommandId=fs_type_response['Command']['CommandId'],
            InstanceId=instance_id
        )
        fs_type = fs_type_invocation['StandardOutputContent'].strip()
        
        # Run appropriate resize command
        if fs_type == 'xfs':
            resize_cmd = f"xfs_growfs {mount_point}"
        elif fs_type in ['ext2', 'ext3', 'ext4']:
            resize_cmd = f"resize2fs {device}"
        else:
            print(f"Unsupported filesystem type: {fs_type}. Skipping resize.")
            return False
        
        # Execute resize
        resize_response = ssm_client.send_command(
            InstanceIds=[instance_id],
            DocumentName='AWS-RunShellScript',
            Parameters={'commands': [resize_cmd]}
        )
        resize_invocation = ssm_client.get_command_invocation(
            CommandId=resize_response['Command']['CommandId'],
            InstanceId=instance_id
        )
        
        if resize_invocation['Status'] == 'Success':
            print(f"Filesystem resize completed successfully for {device}")
            return True
        else:
            print(f"Filesystem resize failed: {resize_invocation['StandardErrorContent']}")
            return False
    except ClientError as e:
        print(f"Error resizing filesystem: {e.response['Error']['Message']}")
        return False

def main():
    # Configuration
    REGION = 'eu-central-1'
    INSTANCE_ID = 'xxxxxxxxxxxxxxxxx'  # Replace with your instance ID
    INCREASE_VOLUME_BY_GB = 10  # Adjust how much to add to the volume
    USAGE_THRESHOLD = 80  # Resize when usage exceeds this percentage (remaining <20%)
    
    # Initialize clients
    ec2_resource = boto3.resource('ec2', region_name=REGION)
    ec2_client = boto3.client('ec2', region_name=REGION)
    
    # Get instance and disk data
    instance = ec2_resource.Instance(INSTANCE_ID)
    disk_usage = get_disk_usage(INSTANCE_ID, REGION)
    if not disk_usage:
        print("Failed to retrieve disk usage. Exiting.")
        return
    
    volume_device_map = get_volume_device_mapping(instance)
    
    # Process each volume
    for volume in instance.volumes.all():
        volume_id = volume.id
        attached_device = volume_device_map.get(volume_id)
        if not attached_device:
            print(f"Volume {volume_id} is not attached to the instance. Skipping.")
            continue
        
        # Match EC2 device to df device (handle partitions like /dev/xvda1 vs /dev/xvda)
        df_device = None
        for dev in disk_usage.keys():
            if dev.startswith(attached_device) or attached_device.startswith(dev):
                df_device = dev
                break
        
        if not df_device:
            print(f"Could not find matching device for {volume_id} ({attached_device}) in disk data. Skipping.")
            continue
        
        current_usage = disk_usage[df_device]['usage_percent']
        mount_point = disk_usage[df_device]['mount_point']
        current_size = volume.size
        
        print(f"\nChecking Volume {volume_id} ({df_device}, {mount_point}):")
        print(f"Current usage: {current_usage}% | Current size: {current_size}GB")
        
        if current_usage >= USAGE_THRESHOLD:
            new_size = current_size + INCREASE_VOLUME_BY_GB
            print(f"Usage exceeds threshold. Resizing to {new_size}GB...")
            
            try:
                # Modify EC2 volume size
                ec2_client.modify_volume(
                    VolumeId=volume_id,
                    Size=new_size,
                    DryRun=False
                )
                print(f"Volume resize initiated. Waiting for completion...")
                
                # Wait for volume modification to finish
                waiter = ec2_client.get_waiter('volume_modification_completed')
                waiter.wait(VolumeIds=[volume_id])
                
                # Resize filesystem on instance
                resize_filesystem(INSTANCE_ID, REGION, df_device, mount_point)
                
            except ClientError as e:
                print(f"Error modifying volume {volume_id}: {e.response['Error']['Message']}")
        else:
            print("Sufficient free space. No action needed.")

if __name__ == "__main__":
    main()

Key Improvements Over Your Initial Code #

1. Filesystem Usage Check: Uses SSM Run Command to execute df -P on the instance, giving accurate free space stats the EC2 API can’t provide.
2. Volume-OS Device Mapping: Handles cases where EC2’s attached device name (e.g., /dev/xvda) differs from the OS’s device name (e.g., /dev/xvda1).
3. Filesystem Resizing: Automatically detects the filesystem type (XFS/ext4) and runs the correct resize command after expanding the volume.
4. Error Handling: Includes try/except blocks for AWS API calls and command execution to catch and report issues.
5. Waiters: Uses AWS waiters to ensure volume modification completes before resizing the filesystem, avoiding race conditions.

Customization Tips #

• Adjust INCREASE_VOLUME_BY_GB to change how much you add to the volume (or modify the logic to resize to a target usage percentage instead of fixed GB).
• Modify USAGE_THRESHOLD if you want to trigger resizing at a different free space level.
• Add logging instead of print statements for production use.

内容的提问来源于stack exchange，提问作者dice2011