基于DynamoDB的AWS事件存储设计：事件重放难题求助

Great question—your core DynamoDB Event Store design looks really solid: using (StreamId, EventId) as the composite key aligns perfectly with aggregate-per-event-stream patterns, conditional writes for optimistic locking are exactly the right call, and DynamoDB Streams + Lambda for event publishing is a standard, reliable approach.

The challenge of replaying all events across thousands of streams is a common one with this pattern, and your initial idea of storing a StreamId array has two key flaws: DynamoDB's 400KB item size limit (which will become a problem as your stream count grows) and race conditions where new streams created mid-replay won't be added to the array until you update it. Let's break down better alternatives:

1. Add a Global Secondary Index (GSI) for Stream Discovery #

This is the most scalable, low-maintenance solution. Here's how to implement it:

• Add a new attribute to every event record (or just to a single "stream metadata" record per stream) like StreamIndexKey with a fixed value, e.g., ALL_STREAMS.
• Create a GSI where:
  • Partition Key = StreamIndexKey
  • Sort Key = StreamId
• For every new stream (when you write the first event for an aggregate), you can either:
  • Include the StreamIndexKey in that first event record, or
  • Write a dedicated metadata record (with Partition Key = StreamId, Sort Key = STREAM_METADATA) that includes StreamIndexKey.

When you need to replay all events:

1. Query the GSI with PartitionKey = ALL_STREAMS to get every unique StreamId (DynamoDB will automatically deduplicate since each stream will have at least one entry in the GSI).
2. For each StreamId, query the main table to fetch all events in order using PartitionKey = ${streamId} (since your Sort Key is EventId, results will be returned in sequence).

This approach automatically includes new streams as they're created—no manual updates needed, and it leverages DynamoDB's native indexing for fast, consistent queries.

2. Use DynamoDB Streams to Build a Centralized Event Log #

If you need to replay events in global order (not per-stream), or want a more efficient full-replay mechanism:

• Configure your existing DynamoDB Stream to send all event writes to a Kinesis Data Stream or an S3 bucket (using Lambda as the intermediary).
• In S3, store events in time-partitioned objects (e.g., s3://your-event-store/year=2024/month=05/day=20/hour=12/events.json).

For full replay, you can simply iterate over the S3 objects in chronological order and process each event. This avoids having to query thousands of individual streams, and it also gives you a durable, immutable copy of all events for disaster recovery.

3. Periodic Stream Inventory Scan (For Smaller Scale) #

If you can't modify your table structure right now, a temporary workaround is to run a periodic Lambda function that:

• Scans your main Event Store table to extract all unique StreamId values.
• Stores these StreamIds in a dedicated "stream inventory" table (e.g., Partition Key = STREAM_INVENTORY, Sort Key = StreamId).

When you need to replay, query the inventory table to get all streams. Note that this approach has tradeoffs:

• Scans are expensive and slow for large tables.
• There's a window where new streams created after the scan won't be included in the replay.

Why Your Initial Array Idea Isn't Ideal #

As you noticed, storing all StreamIds in a single array has two critical issues:

1. Size Limits: DynamoDB items can't exceed 400KB—once you have thousands of streams, this array will hit the limit.
2. Concurrency Conflicts: Updating the array every time a new stream is created requires conditional writes to avoid overwrites, which adds complexity and potential bottlenecks.

内容的提问来源于stack exchange，提问作者Christian Paesante