Amazon S3 API：如何通过单次请求获取指定“文件夹”下完整对象

Hey there! Great question—this is a common optimization folks look for when working with S3's pseudo-folder structure, and the good news is you absolutely can use your existing prefix/delimiter logic to get the results you want in a single HTTP call (or paginated calls for larger datasets). Let’s break this down based on what you’re trying to retrieve:

1. If you only need object metadata (keys, sizes, timestamps, etc.) #

Your current ListObjectsRequest setup is already doing this efficiently! When you use withPrefix(folderPath) and withDelimiter(DELIMITER):

• A single s3Client.listObjects(req) call sends one HTTP request to S3.
• The response returns:
  • Contents: All direct objects in your target "folder" (no extra calls needed).
  • CommonPrefixes: The sub-"folders" under your target path (if any).

For small datasets (≤1000 objects, which is S3's default max per list response), this gets you everything you need in one go. If you have more than 1000 objects, you’ll need to paginate using the nextMarker (for the v1 API) or continuationToken (for the recommended v2 listObjectsV2 API), but each page is still a single HTTP call.

Here’s how your existing code works for this use case (with a v2 update for good measure):

// V1 API (your current setup)
ListObjectsRequest req = new ListObjectsRequest()
    .withBucketName(bucket)
    .withPrefix(folderPath)
    .withDelimiter(DELIMITER);
ObjectListing listing = s3Client.listObjects(req);

// Recommended V2 API (more modern, better pagination)
ListObjectsV2Request reqV2 = ListObjectsV2Request.builder()
    .bucket(bucket)
    .prefix(folderPath)
    .delimiter(DELIMITER)
    .build();
ListObjectsV2Response responseV2 = s3Client.listObjectsV2(reqV2);

If you want all objects under the prefix (including those in sub-folders), just remove the delimiter parameter. This will return every object with the target prefix, regardless of how deep it is in the pseudo-folder hierarchy—still in a single HTTP call for small datasets.

2. If you need the actual content of the objects #

Unfortunately, S3 doesn’t offer a single HTTP call to fetch the content of multiple objects. You’ll still need to iterate over the keys returned from the list call and fetch each object individually with s3Client.getObject().

That said, you can optimize this process:

• Use parallel requests: Spin up a thread pool to fetch multiple objects at the same time (instead of waiting for one to finish before starting the next).
• Use AWS SDK v2’s async clients: These are designed for concurrent operations and can make this pattern much cleaner.

Quick recap #

• Metadata/list only: Your existing approach is already efficient for small datasets—single HTTP call, no extra steps needed.
• Object content: No single-call shortcut, but parallelization will get you better performance than sequential calls.

内容的提问来源于stack exchange，提问作者Azimuts