为超20亿条记录的DynamoDB表添加TTL列的可行方案咨询

Great question—handling TTL for a massive DynamoDB table (2B+ records) requires careful planning to avoid throttling, excessive costs, or long runtimes. Let’s break down your options clearly:

Out-of-the-Box & Managed Solutions #

While there’s no single "one-click" tool to retroactively add TTL to every record in a huge table, AWS offers managed services tailored for large-scale data tasks:

• AWS Glue ETL Jobs: Glue is built for big data processing and integrates natively with DynamoDB. You can create a Glue Job that scans your entire table, calculates the appropriate TTL timestamp (convert your desired expiration date to Unix epoch seconds), and writes updated records back. Glue handles scaling automatically, so you don’t have to manage clusters yourself. Just be sure to configure enough workers for the volume, and temporarily adjust DynamoDB’s read/write capacity to avoid throttling.
• DynamoDB Streams + Lambda (Not Ideal for 2B Records): Lambda works for small to medium datasets, but it’s impractical here. Lambda has concurrency limits, and processing 2B records would drag on for ages while triggering constant throttling. Reserve this for incremental updates after you’ve handled the bulk of your data.

EMR: A Fully Viable Path #

Yes, EMR is absolutely a strong choice for this scenario—especially if you’re already comfortable with big data frameworks like Spark or Hive. Here’s how to approach it:

1. Spin up an EMR Cluster: Pick a configuration with enough core and task nodes to handle the data volume. Spark is recommended for its speed and efficient DynamoDB integration.
2. Read the DynamoDB Table: Use the dynamodb input format for Spark/Hive to scan the table. Optimize reads by enabling parallel scans (splitting the table into segments) to leverage multiple workers at once.
3. Calculate TTL Values: In your Spark job, compute the Unix epoch timestamp for when each record should expire (e.g., 90 days from creation, or a fixed future date).
4. Write Updated Records Back: Use the dynamodb output format to batch-write records with the new TTL attribute. Stick to batch operations (BatchWriteItem) to minimize API calls and avoid throttling.
5. Tune DynamoDB Capacity: To prevent throttling during the update, temporarily switch your table to On-Demand Mode (if it’s not already) or provision higher read/write capacity. You can scale back once the job finishes.

Critical Considerations #

• Enable TTL First: Before updating records, make sure you’ve enabled TTL on your DynamoDB table via the AWS Console or CLI, specifying your chosen TTL attribute name (e.g., ttl).
• Test with a Small Dataset: Always run a test job on a subset of your data first to validate that TTL values are calculated correctly and updates work as expected.
• Cost Management: Scanning and writing 2B records will consume significant DynamoDB capacity. Use Cost Explorer to estimate costs, and consider running the job during off-peak hours if possible.
• Incremental Updates: After handling the bulk of your data, set up a process (like Lambda + Streams) to add TTL to new records as they’re inserted.

内容的提问来源于stack exchange，提问作者Vivek Goel