如何通过编程方式降低/控制Amazon S3请求速率？

问题背景 #

遇到S3返回的503 SlowDown错误，错误栈如下：

Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown; Request ID: TZJYPCC02W0YJFNC; S3 Extended Request ID: uEBpUJSvQ6t9+3rxq7BY70xYPWQeQBltI7BYcEzg9Wb8/rWuENBSaxG5CxyGJBzF+luhvcldIELSITu74/2HNw==; Proxy: null), S3 Extended Request ID: uEBpUJSvQ6t9+3rxq7BY70xYPWQeQBltI7BYcEzg9Wb8/rWuENBSaxG5CxyGJBzF+luhvcldIELSITu74/2HNw==
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1403)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1372)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5437)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5384)
    at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)
    at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6508)
    at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1856)
    at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1816)
    at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:3982)
    ...
    at java.base/java.util.Spliterator.forEachRemaining(Unknown Source)
    at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
    at java.base/java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source)
    at java.base/java.util.concurrent.CountedCompleter.exec(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)

以下是几种编程层面控制请求速率、解决该错误的方法：

1. 配置AWS SDK内置的重试与退避策略 #

AWS Java SDK自带指数退避机制，可针对SlowDown错误调整重试参数，让SDK自动处理请求放缓：

ClientConfiguration clientConfig = new ClientConfiguration();
// 设置最大重试次数，针对503错误会自动触发退避
clientConfig.setMaxErrorRetry(5);
// 自定义重试策略，优先处理限流类错误
RetryPolicy retryPolicy = new RetryPolicy(
    (originalRequest, exception, retryIndex) -> 
        exception instanceof AmazonS3Exception && 
        "SlowDown".equals(((AmazonS3Exception) exception).getErrorCode()),
    new ExponentialBackoffStrategy(1000, 5000),
    5,
    true
);
clientConfig.setRetryPolicy(retryPolicy);

AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
    .withClientConfiguration(clientConfig)
    .build();

指数退避会在每次重试时递增等待时间，避免持续触发S3限流机制。

2. 实现自定义请求限流 #

通过信号量或速率限制器，直接控制并发请求数或每秒请求量：

方式一：用Semaphore限制并发数 #

// 限制同时最多10个S3请求
Semaphore semaphore = new Semaphore(10);

// 发起请求前获取许可
semaphore.acquire();
try {
    s3Client.putObject(bucketName, key, file);
} finally {
    // 请求完成后释放许可
    semaphore.release();
}

方式二：用Guava RateLimiter控制请求速率 #

// 限制每秒最多20个请求
RateLimiter rateLimiter = RateLimiter.create(20.0);

// 等待获取请求许可
rateLimiter.acquire();
s3Client.putObject(bucketName, key, file);

3. 批量操作减少请求次数 #

将多个单个请求合并为批量操作，降低总请求量：

• 删除操作使用deleteObjects批量删除多个对象；
• 小文件可合并为压缩包后上传（需业务逻辑允许）；
• 大文件使用S3分段上传（Multipart Upload），SDK会自动拆分并控制分段请求的并发数。

示例：批量删除对象

DeleteObjectsRequest deleteRequest = new DeleteObjectsRequest(bucketName);
List<KeyVersion> keysToDelete = Arrays.asList(
    new KeyVersion("file1.txt"),
    new KeyVersion("file2.jpg")
);
deleteRequest.setKeys(keysToDelete);
s3Client.deleteObjects(deleteRequest);

4. 优化异步请求的线程池配置 #

如果使用异步S3客户端，需合理配置线程池大小，避免线程过载导致请求量超标：

// 固定大小线程池，控制并发请求数
ExecutorService executor = Executors.newFixedThreadPool(10);
AmazonS3Async s3AsyncClient = AmazonS3AsyncClientBuilder.standard()
    .withExecutorFactory(() -> executor)
    .build();

// 提交异步请求并处理结果
s3AsyncClient.putObjectAsync(bucketName, key, file)
    .whenComplete((result, exception) -> {
        if (exception instanceof AmazonS3Exception && "SlowDown".equals(((AmazonS3Exception) exception).getErrorCode())) {
            // 针对限流错误手动重试或记录日志
        }
    });

5. 捕获SlowDown错误后主动退避 #

在业务代码中捕获限流错误，主动添加延迟后重试：

int retryCount = 0;
boolean success = false;
while (!success && retryCount < 3) {
    try {
        s3Client.putObject(bucketName, key, file);
        success = true;
    } catch (AmazonS3Exception e) {
        if ("SlowDown".equals(e.getErrorCode()) && retryCount < 3) {
            retryCount++;
            // 指数退避等待
            Thread.sleep((long) Math.pow(2, retryCount) * 1000);
        } else {
            throw e;
        }
    }
}

内容的提问来源于stack exchange，提问作者Mykhaylo Adamovych