AWS环境下Lambda定时获取数据并存储JSON的服务选型咨询

Hey Tony, based on your requirement to automatically fetch API data daily, store it as a JSON array, and perform CRUD operations via Lambda, here's a breakdown of the ideal AWS services to use:

1. Scheduled Trigger: Amazon EventBridge #

• Use EventBridge Rules with a cron expression (e.g., 0 5 * * ? * to run daily at 5 AM UTC) to trigger your Lambda function on schedule. It’s fully managed, reliable, and integrates seamlessly with Lambda—no servers to maintain.

2. Core Execution: AWS Lambda #

• Deploy two focused Lambda functions (or reuse one with logic branching) to handle your workflow:
  • Daily Data Fetch: Write code in your preferred runtime (Python, Node.js, etc.) to call the 20-30 APIs, aggregate responses into a JSON array, and save the result to your chosen storage.
  • CRUD Operations: Build functions to read, update, delete, or add entries to your stored JSON data. Lambda is serverless, scales automatically, and fits perfectly with event-driven tasks like this.

3. Storage Options (Pick Based on Your CRUD Needs) #

Option A: Amazon S3 (For Whole JSON File Operations) #

• Ideal if you plan to work with the entire JSON array as a single object (e.g., replace the whole file, read the full array for bulk processing).
• Benefits: Low cost, high durability, and straightforward integration with Lambda. Store files with a structured key like daily-api-data/YYYY-MM-DD.json to track historical versions (enable S3 Versioning if you need to retain old copies).

Option B: Amazon DynamoDB (For Granular Entry-Level CRUD) #

• Perfect if you need to modify individual entries within the JSON array (e.g., update a specific API’s response, delete an outdated entry).
• Structure each API response as a separate item in a DynamoDB table (use a unique key like apiId or timestamp). Lambda can interact directly with DynamoDB via AWS SDKs for fast, item-level operations. DynamoDB is serverless, auto-scales, and offers low-latency access.

4. Security & Observability: IAM + CloudWatch Logs #

• IAM Roles: Create a dedicated IAM role for your Lambda functions with minimal permissions (e.g., s3:PutObject/s3:GetObject for S3, dynamodb:PutItem/dynamodb:GetItem for DynamoDB). Avoid hardcoding credentials—always use IAM for secure access.
• Amazon CloudWatch Logs: Automatically captures Lambda execution logs, making it easy to debug API fetch failures, storage errors, or CRUD operation issues. You can also set up CloudWatch Alarms to get notified if the daily fetch task fails.

Quick Workflow Recap #

1. EventBridge triggers the fetch Lambda at your scheduled morning time.
2. Fetch Lambda calls APIs, aggregates data into a JSON array, and saves it to S3 or DynamoDB.
3. Your CRUD Lambda functions interact with the stored data as needed, using IAM permissions for secure access.
4. CloudWatch Logs monitors all activity for easy troubleshooting.

内容的提问来源于stack exchange，提问作者Tony