通过获取VPC流量日志估算GuardDuty月度成本

Got it, let's figure out how to get the VPC Flow Logs data volume you need for GuardDuty cost estimation. Since you already used boto3 to count CloudTrail logs from S3, here are a few straightforward methods tailored to your workflow:

Method 1: Directly Calculate Total Size from S3 Bucket #

If your VPC Flow Logs are stored directly in an S3 bucket (the most common setup), you can use boto3 to iterate through all log objects and sum their sizes. This gives you the exact GB volume GuardDuty will use for pricing.

Here's a reusable script:

import boto3

def get_vpc_logs_total_gb(bucket_name, log_prefix=""):
    s3_client = boto3.client('s3')
    total_bytes = 0
    paginator = s3_client.get_paginator('list_objects_v2')

    # Paginate through all objects in the bucket/prefix to avoid size limits
    for page in paginator.paginate(Bucket=bucket_name, Prefix=log_prefix):
        if 'Contents' in page:
            for obj in page['Contents']:
                total_bytes += obj['Size']

    # Convert bytes to gigabytes (using binary conversion: 1 GB = 1024^3 bytes)
    total_gb = total_bytes / (1024 ** 3)
    return round(total_gb, 2)

# Example usage: Replace with your bucket and log prefix
vpc_log_bucket = "your-vpc-flow-logs-bucket"
vpc_log_prefix = "AWSLogs/123456789012/vpcflowlogs/us-east-1/"
print(f"Total monthly VPC Flow Logs size: {get_vpc_logs_total_gb(vpc_log_bucket, vpc_log_prefix)} GB")

Tip: If your logs are split across multiple prefixes (e.g., per region), just call this function for each prefix and sum the results.

Method 2: Use AWS Cost Explorer for Monthly Storage Data #

If you want a quicker way to get historical monthly storage volume (without writing loop code), Cost Explorer can pull the exact GB-months used by your VPC log bucket. This matches the metric GuardDuty uses for pricing.

Here's how to do it with boto3:

import boto3

def get_monthly_s3_storage_usage(bucket_name, start_date, end_date):
    ce_client = boto3.client('cost-explorer')
    response = ce_client.get_cost_and_usage(
        TimePeriod={'Start': start_date, 'End': end_date},
        Granularity='MONTHLY',
        Metrics=['UsageQuantity'],
        Filter={
            'And': [
                {
                    'Dimensions': {
                        'Key': 'S3_BUCKET_NAME',
                        'Values': [bucket_name]
                    }
                },
                {
                    'Dimensions': {
                        'Key': 'USAGE_TYPE_GROUP',
                        'Values': ['S3 Storage']
                    }
                }
            ]
        }
    )
    # Extract the total usage quantity (in GB-months)
    total_gb = float(response['ResultsByTime'][0]['Total']['UsageQuantity']['Amount'])
    return round(total_gb, 2)

# Example usage: Replace with your dates (YYYY-MM-DD)
print(f"Monthly VPC Log storage: {get_monthly_s3_storage_usage('your-vpc-flow-logs-bucket', '2024-04-01', '2024-05-01')} GB")

Note: You can also do this via the AWS Console by navigating to Cost Explorer, filtering for your bucket, and selecting "Usage Quantity" as the metric.

Method 3: Pull Data from CloudWatch Logs (If Using Intermediate Logging) #

If your VPC Flow Logs are first sent to CloudWatch Logs before being archived to S3, you can use CloudWatch Metrics to get the total incoming bytes over the month. This is useful if you want to verify the volume matches what's stored in S3.

Here's a script to fetch that data:

import boto3
from datetime import datetime, timedelta

def get_cloudwatch_logs_monthly_volume(log_group_name):
    cw_client = boto3.client('cloudwatch')
    end_time = datetime.now()
    start_time = end_time - timedelta(days=30)  # Last 30 days

    response = cw_client.get_metric_statistics(
        Namespace='AWS/Logs',
        MetricName='IncomingBytes',
        Dimensions=[{'Name': 'LogGroupName', 'Value': log_group_name}],
        StartTime=start_time,
        EndTime=end_time,
        Period=86400,  # Aggregate daily
        Statistics=['Sum']
    )

    total_bytes = sum(datapoint['Sum'] for datapoint in response['Datapoints'])
    total_gb = total_bytes / (1024 ** 3)
    return round(total_gb, 2)

# Example usage
print(f"Monthly VPC Logs from CloudWatch: {get_cloudwatch_logs_monthly_volume('your-vpc-log-group')} GB")

Quick Recommendation #

If your logs are directly stored in S3, Method 1 is the most accurate for GuardDuty pricing (since GuardDuty uses the actual stored size in S3). Methods 2 and 3 are great for cross-verification or if you prefer using AWS managed metrics.

内容的提问来源于stack exchange，提问作者Nick Ruha