如何用s3.Object直接在S3生成并写入AWS账单CSV文件？

Absolutely! You can absolutely use s3.Object to create and populate a CSV file directly in Amazon S3—no need to save it locally first. Let’s figure out why your initial attempt didn’t generate a file, then walk through a complete working solution that covers both creating a new CSV in S3 and appending content to an existing one.

Why Your Initial Attempt Might Have Failed #

Before diving into code, here are the most common reasons you might see no errors but no file in S3:

• Incorrect bucket/key name: Double-check that your bucket name is spelled correctly and the S3 key (file path) doesn’t have invalid characters or leading slashes (e.g., use billing-reports/account-app-costs.csv instead of /billing-reports/account-app-costs.csv).
• Missing IAM permissions: The role/user running your code needs the s3:PutObject permission for the target bucket.
• Empty content: If your JSON data was empty or you didn’t properly populate the CSV buffer, S3 might create an empty file (or skip it if your code doesn’t handle empty content).
• Forgot to call put(): It’s easy to set up the s3.Object but forget to execute the upload with the put() method.

Complete Code Example: Create CSV Directly in S3 #

Here’s how to take your Cost Explorer JSON output, convert it to CSV, and upload it straight to S3 using s3.Object:

import boto3
import csv
from io import StringIO

# Initialize clients
ce_client = boto3.client('ce')
s3 = boto3.resource('s3')

# Step 1: Get your Cost Explorer data (replace with your actual query)
response = ce_client.get_cost_and_usage(
    TimePeriod={
        'Start': '2024-01-01',
        'End': '2024-01-31'
    },
    Granularity='MONTHLY',
    Metrics=['UnblendedCost'],
    GroupBy=[
        {'Type': 'DIMENSION', 'Key': 'LINKED_ACCOUNT'},
        {'Type': 'TAG', 'Key': 'Application'}  # Assuming you use an Application tag
    ]
)

# Step 2: Parse the JSON into rows for CSV
json_data = response['ResultsByTime'][0]['Groups']
csv_rows = []
# Extract header from the first group's keys + metric
headers = ['LinkedAccount', 'Application', 'UnblendedCost']
csv_rows.append(headers)

for group in json_data:
    linked_account = group['Keys'][0]
    application = group['Keys'][1] if len(group['Keys']) > 1 else 'Uncategorized'
    cost = group['Metrics']['UnblendedCost']['Amount']
    csv_rows.append([linked_account, application, cost])

# Step 3: Build CSV in memory (no local file)
csv_buffer = StringIO()
csv_writer = csv.writer(csv_buffer)
csv_writer.writerows(csv_rows)

# Step 4: Upload to S3 using s3.Object
bucket_name = 'your-bucket-name'
s3_key = 'billing-reports/account-app-costs.csv'  # Path in S3

try:
    s3_object = s3.Object(bucket_name, s3_key)
    s3_object.put(
        Body=csv_buffer.getvalue(),
        ContentType='text/csv',
        ContentEncoding='utf-8'
    )
    print(f"Successfully created CSV at s3://{bucket_name}/{s3_key}")
except Exception as e:
    print(f"Error uploading to S3: {str(e)}")

Append Content to an Existing S3 CSV #

Since S3 is an object store (not a file system), you can’t "open" a file in write mode like you would locally. Instead, you’ll download the existing CSV to memory, append your new rows, then re-upload it:

def append_to_s3_csv(bucket_name, s3_key, new_rows):
    s3 = boto3.resource('s3')
    s3_object = s3.Object(bucket_name, s3_key)
    
    try:
        # Download existing CSV content
        existing_content = s3_object.get()['Body'].read().decode('utf-8')
        csv_buffer = StringIO(existing_content)
        csv_reader = csv.reader(csv_buffer)
        existing_rows = list(csv_reader)
        
        # Append new rows (skip header if new_rows includes it)
        existing_rows.extend(new_rows)
        
        # Write updated content back to S3
        updated_buffer = StringIO()
        csv_writer = csv.writer(updated_buffer)
        csv_writer.writerows(existing_rows)
        
        s3_object.put(
            Body=updated_buffer.getvalue(),
            ContentType='text/csv',
            ContentEncoding='utf-8'
        )
        print(f"Successfully appended to CSV at s3://{bucket_name}/{s3_key}")
    except s3_object.client.exceptions.NoSuchKey:
        # If file doesn't exist, create it with new rows
        print("File not found, creating new CSV...")
        csv_buffer = StringIO()
        csv_writer = csv.writer(csv_buffer)
        csv_writer.writerows(new_rows)
        s3_object.put(
            Body=csv_buffer.getvalue(),
            ContentType='text/csv',
            ContentEncoding='utf-8'
        )

# Example usage: append new cost rows
new_cost_rows = [
    ['123456789012', 'MyNewApp', '45.67'],
    ['987654321098', 'AnotherApp', '12.34']
]
append_to_s3_csv('your-bucket-name', 'billing-reports/account-app-costs.csv', new_cost_rows)

Key Notes #

• Permissions: Ensure your IAM role/user has s3:PutObject (for creating/overwriting) and s3:GetObject (for appending) permissions on the target bucket.
• Memory Considerations: If your CSV is extremely large, using StringIO might not be ideal—consider using S3 Multipart Upload instead, but for most billing reports, in-memory handling is fine.
• Encoding: Always specify utf-8 encoding to avoid character issues in your CSV.

内容的提问来源于stack exchange，提问作者Raj