如何使用AWS SDK V2 for Java复制大于5GB的对象

S3中单个对象超过5GB时，必须通过**分段复制（Multipart Copy）**完成操作，以下是基于AWS SDK V2 for Java的具体实现方案：

核心步骤与代码示例 #

1. 初始化分段复制任务 #

首先创建分段上传请求，获取唯一的uploadId，作为后续所有分段操作的标识：

import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.CreateMultipartUploadRequest;
import software.amazon.awssdk.services.s3.model.CreateMultipartUploadResponse;

public class S3MultipartCopy {
    private static final S3Client s3Client = S3Client.create();

    public static String initMultipartCopy(String sourceBucket, String sourceKey, String targetBucket, String targetKey) {
        CreateMultipartUploadRequest request = CreateMultipartUploadRequest.builder()
                .bucket(targetBucket)
                .key(targetKey)
                .build();

        CreateMultipartUploadResponse response = s3Client.createMultipartUpload(request);
        return response.uploadId();
    }
}

2. 分段复制对象 #

计算分段大小（建议设置为1GB~5GB，示例中用1GB），遍历每个分段，发起复制请求并记录每个分段的ETag：

import software.amazon.awssdk.services.s3.model.UploadPartCopyRequest;
import software.amazon.awssdk.services.s3.model.UploadPartCopyResponse;
import software.amazon.awssdk.services.s3.model.CompletedPart;
import java.util.ArrayList;
import java.util.List;

public static List<CompletedPart> copyParts(String sourceBucket, String sourceKey, String targetBucket, String targetKey, String uploadId) {
    // 获取源对象大小
    long objectSize = s3Client.headObject(req -> req.bucket(sourceBucket).key(sourceKey)).contentLength();
    // 设置分段大小为1GB
    long partSize = 1024 * 1024 * 1024;
    int partCount = (int) Math.ceil((double) objectSize / partSize);
    List<CompletedPart> completedParts = new ArrayList<>();

    for (int partNumber = 1; partNumber <= partCount; partNumber++) {
        long start = (partNumber - 1) * partSize;
        long end = Math.min(start + partSize - 1, objectSize - 1);

        UploadPartCopyRequest copyRequest = UploadPartCopyRequest.builder()
                .sourceBucket(sourceBucket)
                .sourceKey(sourceKey)
                .bucket(targetBucket)
                .key(targetKey)
                .uploadId(uploadId)
                .partNumber(partNumber)
                .copySourceRange(String.format("bytes=%d-%d", start, end))
                .build();

        UploadPartCopyResponse copyResponse = s3Client.uploadPartCopy(copyRequest);
        completedParts.add(CompletedPart.builder()
                .partNumber(partNumber)
                .eTag(copyResponse.copyPartResult().eTag())
                .build());
    }
    return completedParts;
}

3. 完成分段复制 #

收集所有分段的ETag信息，发起完成请求，结束整个复制任务：

import software.amazon.awssdk.services.s3.model.CompleteMultipartUploadRequest;
import software.amazon.awssdk.services.s3.model.CompletedMultipartUpload;

public static void completeMultipartCopy(String targetBucket, String targetKey, String uploadId, List<CompletedPart> completedParts) {
    CompletedMultipartUpload completedUpload = CompletedMultipartUpload.builder()
            .parts(completedParts)
            .build();

    CompleteMultipartUploadRequest completeRequest = CompleteMultipartUploadRequest.builder()
            .bucket(targetBucket)
            .key(targetKey)
            .uploadId(uploadId)
            .multipartUpload(completedUpload)
            .build();

    s3Client.completeMultipartUpload(completeRequest);
}

4. 异常处理与任务中止 #

如果复制过程中出现异常，需要及时中止分段任务，避免冗余分段占用存储：

import software.amazon.awssdk.services.s3.model.AbortMultipartUploadRequest;

public static void abortMultipartCopy(String targetBucket, String targetKey, String uploadId) {
    AbortMultipartUploadRequest abortRequest = AbortMultipartUploadRequest.builder()
            .bucket(targetBucket)
            .key(targetKey)
            .uploadId(uploadId)
            .build();
    s3Client.abortMultipartUpload(abortRequest);
}

调用示例 #

public static void main(String[] args) {
    String sourceBucket = "source-bucket-name";
    String sourceKey = "large-object-key";
    String targetBucket = "target-bucket-name";
    String targetKey = "copied-large-object-key";

    String uploadId = null;
    try {
        uploadId = initMultipartCopy(sourceBucket, sourceKey, targetBucket, targetKey);
        List<CompletedPart> completedParts = copyParts(sourceBucket, sourceKey, targetBucket, targetKey, uploadId);
        completeMultipartCopy(targetBucket, targetKey, uploadId, completedParts);
        System.out.println("大对象复制完成");
    } catch (Exception e) {
        if (uploadId != null) {
            abortMultipartCopy(targetBucket, targetKey, uploadId);
            System.out.println("复制失败，已中止分段任务");
        }
        e.printStackTrace();
    } finally {
        s3Client.close();
    }
}

注意事项 #

• 分段大小建议设置在1GB到5GB之间，过小会增加API请求次数，过大可能降低重试容错性
• 可通过线程池实现多分段并行复制，提升复制效率
• 确保S3Client拥有源桶的S3:GetObject权限和目标桶的S3:PutObject、S3:AbortMultipartUpload等相关权限

内容的提问来源于stack exchange，提问作者Hardik