如何使用Java SDK在AWS S3上动态构建Zip归档文件？

核心思路 #

利用**S3分段上传（Multipart Upload）**特性，配合Java的ZipOutputStream流式生成Zip内容，全程无需将整个Zip文件缓存到内存或磁盘：

1. 先在S3中初始化分段上传，生成一个Upload ID作为Zip的"占位符"；
2. 流式遍历目标S3对象，将每个对象的内容实时写入ZipOutputStream；
3. 每积累到固定大小（推荐5MB，符合S3分段上传的最小要求）的Zip数据，就上传一个分段到S3；
4. 所有对象处理完成后，合并所有分段生成最终的Zip文件。

具体实现（基于AWS Java SDK 2.x + Spring） #

1. 依赖准备

确保项目中引入AWS S3 SDK和Spring Web依赖：

<!-- AWS S3 SDK -->
<dependency>
    <groupId>software.amazon.awssdk</groupId>
    <artifactId>s3</artifactId>
</dependency>
<!-- Spring Web -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>

2. 核心代码实现

import software.amazon.awssdk.core.RequestBody;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.*;
import org.springframework.stereotype.Service;

import java.io.*;
import java.util.ArrayList;
import java.util.List;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;

@Service
public class S3ZipService {
    private final S3Client s3Client;
    private static final int PART_SIZE = 5 * 1024 * 1024; // 5MB分段大小
    private static final int BUFFER_SIZE = 8192; // 流处理缓冲大小

    public S3ZipService(S3Client s3Client) {
        this.s3Client = s3Client;
    }

    public void createZipFromS3Objects(String bucketName, List<String> objectKeys, String targetZipKey) {
        // 1. 初始化S3分段上传，创建Zip占位符
        CreateMultipartUploadRequest createRequest = CreateMultipartUploadRequest.builder()
                .bucket(bucketName)
                .key(targetZipKey)
                .contentType("application/zip")
                .build();
        String uploadId = s3Client.createMultipartUpload(createRequest).uploadId();

        List<CompletedPart> completedParts = new ArrayList<>();
        int partNumber = 1;
        byte[] buffer = new byte[BUFFER_SIZE];

        try (ByteArrayOutputStream partStream = new ByteArrayOutputStream(PART_SIZE);
             ZipOutputStream zipOut = new ZipOutputStream(partStream)) {

            zipOut.setLevel(ZipOutputStream.DEFAULT_COMPRESSION); // 设置压缩级别

            // 2. 遍历所有目标对象，流式写入Zip
            for (String objectKey : objectKeys) {
                // 获取S3对象的输入流
                GetObjectRequest getObjectRequest = GetObjectRequest.builder()
                        .bucket(bucketName)
                        .key(objectKey)
                        .build();
                try (InputStream s3ObjectStream = s3Client.getObject(getObjectRequest)) {
                    // 添加Zip条目
                    ZipEntry zipEntry = new ZipEntry(objectKey);
                    zipOut.putNextEntry(zipEntry);

                    // 流式压缩并写入分段流
                    int bytesRead;
                    while ((bytesRead = s3ObjectStream.read(buffer)) != -1) {
                        zipOut.write(buffer, 0, bytesRead);
                        // 达到分段大小则上传
                        if (partStream.size() >= PART_SIZE) {
                            uploadPart(bucketName, targetZipKey, uploadId, partNumber, partStream, completedParts);
                            partNumber++;
                            partStream.reset();
                        }
                    }
                    zipOut.closeEntry();
                }
            }

            // 3. 上传最后一段剩余数据
            if (partStream.size() > 0) {
                uploadPart(bucketName, targetZipKey, uploadId, partNumber, partStream, completedParts);
            }

            // 4. 完成分段上传，合并生成最终Zip
            CompleteMultipartUploadRequest completeRequest = CompleteMultipartUploadRequest.builder()
                    .bucket(bucketName)
                    .key(targetZipKey)
                    .uploadId(uploadId)
                    .multipartUpload(CompletedMultipartUpload.builder().parts(completedParts).build())
                    .build();
            s3Client.completeMultipartUpload(completeRequest);

        } catch (Exception e) {
            // 异常时中止分段上传，清理S3中残留的分段
            AbortMultipartUploadRequest abortRequest = AbortMultipartUploadRequest.builder()
                    .bucket(bucketName)
                    .key(targetZipKey)
                    .uploadId(uploadId)
                    .build();
            s3Client.abortMultipartUpload(abortRequest);
            throw new RuntimeException("S3 Zip构建失败", e);
        }
    }

    // 辅助方法：上传单个分段
    private void uploadPart(String bucketName, String targetZipKey, String uploadId, int partNumber,
                            ByteArrayOutputStream partStream, List<CompletedPart> completedParts) throws IOException {
        UploadPartRequest uploadRequest = UploadPartRequest.builder()
                .bucket(bucketName)
                .key(targetZipKey)
                .uploadId(uploadId)
                .partNumber(partNumber)
                .contentLength((long) partStream.size())
                .build();

        try (InputStream inputStream = new ByteArrayInputStream(partStream.toByteArray())) {
            UploadPartResponse response = s3Client.uploadPart(uploadRequest, RequestBody.fromInputStream(inputStream, partStream.size()));
            completedParts.add(CompletedPart.builder()
                    .partNumber(partNumber)
                    .eTag(response.eTag())
                    .build());
        }
    }
}

关键注意事项 #

• 分段大小设置：S3要求除最后一段外，其他分段大小至少为5MB，避免因分段过小导致API调用次数过多；
• 流资源管理：所有流必须通过try-with-resources自动关闭，避免资源泄漏；
• 异常处理：构建过程中出现异常时，必须调用abortMultipartUpload清理S3中的临时分段，避免产生无效存储占用；
• 性能优化：可通过BufferedInputStream包装S3对象输入流，提升读取效率；也可根据需求调整Zip压缩级别。

内容的提问来源于stack exchange，提问作者kwarter