使用DynamoDB batchLoad时因重复KeyPair报错的问题咨询

First, to cut straight to the point: Yes, batchLoad behaves exactly like batchWrite here — if your input list of KeyPair entries contains duplicates (where both the hash key and range key are identical across entries), you will get the "provided list of item keys contains duplicates" exception.

Let’s cover your questions one by one:

1. Is this intentional design? #

Absolutely. DynamoDB enforces unique keys for all batch operations (read or write) to avoid ambiguous requests. For batch reads, the service can’t assume whether you meant to fetch the same item multiple times by accident, or if that’s an intentional request. Instead of making assumptions that could lead to unexpected behavior or wasted read capacity, DynamoDB fails fast with this error.

2. Is there a configuration to auto-drop duplicate keys? #

Unfortunately, no — there’s no built-in setting (either in the AWS SDK for Java or on the DynamoDB service side) that will automatically remove duplicate KeyPair entries from your batchLoad request. This responsibility falls entirely to the caller.

3. How to fix this? #

You’ll need to ensure your input list only contains unique KeyPair entries before passing it to batchLoad. The easiest way to do this in Java is to leverage a Set, since the SDK’s KeyPair class properly implements equals() and hashCode() to recognize identical hash/range key pairs.

Here’s a quick code example:

// Your original list with potential duplicate keys
List<KeyPair> rawKeys = ...;

// Remove duplicates using a HashSet
Set<KeyPair> uniqueKeySet = new HashSet<>(rawKeys);
List<KeyPair> uniqueKeys = new ArrayList<>(uniqueKeySet);

// Build and execute your batchLoad request
BatchLoadItemRequest batchRequest = BatchLoadItemRequest.builder()
    .requestItems(Map.of(
        "your-table-name",
        KeysAndAttributes.builder()
            .keys(uniqueKeys)
            .build()
    ))
    .build();

BatchLoadItemResponse response = dynamoDbClient.batchLoadItem(batchRequest);

Alternatively, you can manually iterate through the list and track seen keys if you need more control over which duplicates to keep (though the Set approach works for most cases).

A quick note: Pre-emptively removing duplicates isn’t just about avoiding errors — it also saves you wasted read capacity units (RCUs), since DynamoDB will fail the entire batch request as soon as it detects duplicates, rather than processing any of the entries.

内容的提问来源于stack exchange，提问作者Destructor