从DynamoDB导出数据至CSV：Lambda可行性及替代方案咨询

Yes, Lambda is absolutely viable for this task—and the 15-minute timeout is more than enough #

Let’s break down your questions one by one with practical context:

1. Can Lambda be used to export 150k DynamoDB records? #

Absolutely. Lambda integrates seamlessly with DynamoDB via the boto3 SDK, making it a great serverless option for this job. You can use the scan() API (or query() if you have a partition key to filter results efficiently) to fetch records in batches, process them, and export to a file (we recommend storing the final file in S3, since Lambda’s local /tmp storage is limited but more than sufficient for your data size).

For best performance, assign at least 1024MB of memory to your Lambda function—Lambda allocates CPU and network bandwidth proportionally to memory, so more memory will speed up batch processing and API calls.

2. Will the 15-minute Lambda timeout be sufficient? #

This is a no-brainer. Let’s do quick math to prove it:

• Your total data size: 150,000 records × 430 bytes = ~61.5 MB. That’s tiny compared to what Lambda can handle in a single execution.
• Even if you fetch 1,000 records per scan() call (the maximum page size for DynamoDB), you’ll only need 150 API calls. Each call (including data transfer and light processing) might take 1-2 seconds at worst—total execution time would be well under 5 minutes, leaving plenty of room for overhead like retries.

If you’re worried about throttling, add exponential backoff to your boto3 calls, but with 150k records, throttling is unlikely unless your DynamoDB table has extremely low read capacity.

3. Better Python implementation (since UI exports are limited to 100 records) #

Here’s a production-ready Lambda-compatible Python script that exports DynamoDB records to a JSON file in S3. It handles pagination, error handling, and efficient batch processing:

import boto3
import json
from botocore.exceptions import ClientError

# Initialize clients
dynamodb = boto3.resource('dynamodb')
s3 = boto3.client('s3')

# Configuration - update these values
TABLE_NAME = 'your-target-table'
S3_BUCKET = 'your-export-bucket'
EXPORT_FILE_KEY = 'dynamo-exports/weekly-records.json'

def export_dynamodb_records():
    table = dynamodb.Table(TABLE_NAME)
    all_records = []
    last_evaluated_key = None

    # Paginate through all DynamoDB records
    while True:
        scan_params = {
            'Limit': 1000,  # Max batch size to minimize API calls
            # Optional: Use ProjectionExpression to fetch only needed fields
            # 'ProjectionExpression': 'user_id, order_date, total_amount'
        }
        if last_evaluated_key:
            scan_params['ExclusiveStartKey'] = last_evaluated_key
        
        try:
            response = table.scan(**scan_params)
            all_records.extend(response['Items'])
            last_evaluated_key = response.get('LastEvaluatedKey')
            if not last_evaluated_key:
                break
        except ClientError as e:
            print(f"Scan failed: {e.response['Error']['Message']}")
            raise

    # Convert records to JSON and upload to S3
    json_output = json.dumps(all_records, indent=2)
    try:
        s3.put_object(
            Bucket=S3_BUCKET,
            Key=EXPORT_FILE_KEY,
            Body=json_output,
            ContentType='application/json'
        )
        print(f"Successfully exported {len(all_records)} records to S3")
        return len(all_records)
    except ClientError as e:
        print(f"S3 upload failed: {e.response['Error']['Message']}")
        raise

def lambda_handler(event, context):
    try:
        record_count = export_dynamodb_records()
        return {
            'statusCode': 200,
            'body': json.dumps(f"Export completed: {record_count} records saved")
        }
    except Exception as e:
        return {
            'statusCode': 500,
            'body': json.dumps(f"Export failed: {str(e)}")
        }

Key optimizations in this script: #

• Pagination handling: Uses LastEvaluatedKey to ensure no records are missed during the scan.
• Batch processing: Fetches 1000 records per call to minimize API overhead.
• Selective fetching: Uncomment the ProjectionExpression line to only retrieve fields you need, reducing data transfer time and payload size.
• Error handling: Includes try/except blocks for DynamoDB and S3 operations to catch and report issues clearly.

Bonus: AWS-native alternative (no code needed) #

If you don’t want to maintain custom code, use DynamoDB Export to S3—a serverless feature that lets you export entire tables or specific time ranges directly to S3 in Parquet or JSON format. It’s scalable, avoids Lambda execution limits entirely, and works perfectly for your 150k records.

内容的提问来源于stack exchange，提问作者Asfar Irshad