LocalStack 3.0.2 S3连接拒绝：大对象量测试故障排查求助

问题重现 #

在使用TestContainers运行LocalStack 3.0.2，对基于aws-java-sdk的Spring Boot应用做集成测试时，批量生成约100万个结构化路径的S3文本对象，触发连接拒绝异常：

com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to 127.0.0.1:32822 [/127.0.0.1] failed: Connection refused

异常堆栈：

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1219)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1165)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5520)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5467)
at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:422)
at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6601)
at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1891)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1851)

Docker容器日志无报错，故障触发的对象数量不固定，但能稳定处理约95万个对象，需支持200-1500万个对象的测试场景。

原因分析 #

1. AWS SDK连接池耗尽：默认连接池大小有限，批量高并发上传时，新请求无法获取空闲连接，导致连接拒绝。
2. LocalStack资源不足：TestContainers默认给容器分配的内存、CPU配额较低，处理百万级对象时资源耗尽，服务暂时无响应。
3. S3存储后端瓶颈：LocalStack默认使用文件系统存储S3对象，大量对象写入时IO性能不足，导致服务无法及时响应请求。
4. 并发请求过载：无限制的并发上传请求超过LocalStack的处理能力，引发服务端连接队列溢出。

解决办法 #

1. 调优AWS SDK连接池配置 #

增大连接池容量，调整超时和重试策略，避免连接耗尽：

AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
    .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(localStackEndpoint, region))
    .withClientConfiguration(new ClientConfiguration()
        .withMaxConnections(500) // 提升最大连接数
        .withConnectionTimeout(10000) // 延长连接超时时间
        .withSocketTimeout(30000) // 延长Socket读写超时
        .withRetryPolicy(PredefinedRetryPolicies.getDefaultRetryPolicyWithCustomMaxRetries(5))) // 增加重试次数
    .build();

2. 给LocalStack容器分配更多资源 #

在TestContainers配置中增加内存、CPU配额，同时改用内存存储优化性能：

LocalStackContainer localStack = new LocalStackContainer(DockerImageName.parse("localstack/localstack:3.0.2"))
    .withEnv("LOCALSTACK_MEMORY", "8192") // 分配8GB内存
    .withEnv("LOCALSTACK_CPU", "4") // 分配4核CPU
    .withEnv("S3_BACKEND", "memory") // 改用内存存储，大幅提升写入性能（测试数据不持久）
    .withServices(LocalStackContainer.Service.S3);

3. 优化批量上传方式 #

使用TransferManager替代单条putObject调用，它会自动处理并发、分片和重试：

TransferManager transferManager = TransferManagerBuilder.standard()
    .withS3Client(s3Client)
    .withMultipartUploadThreshold((long) (5 * 1024 * 1024)) // 5MB以上自动分片
    .build();

List<Upload> uploadTasks = new ArrayList<>();
for (int i = 0; i < 1500000; i++) {
    String key = "structured/path/" + i + ".txt";
    String content = "sample content " + i;
    Upload task = transferManager.upload(bucketName, key, new ByteArrayInputStream(content.getBytes()), new ObjectMetadata());
    uploadTasks.add(task);
}

// 等待所有上传完成
for (Upload task : uploadTasks) {
    task.waitForCompletion();
}
transferManager.shutdownNow();

4. 控制并发请求数 #

用线程池限制并发上传的数量，避免LocalStack过载：

ExecutorService executor = Executors.newFixedThreadPool(50); // 限制50个并发线程
List<Future<?>> futures = new ArrayList<>();

for (int i = 0; i < 1500000; i++) {
    int finalIndex = i;
    futures.add(executor.submit(() -> {
        String key = "path/" + finalIndex + ".txt";
        s3Client.putObject(bucketName, key, "test content " + finalIndex);
    }));
}

// 等待所有任务完成
for (Future<?> future : futures) {
    future.get();
}
executor.shutdown();

5. 延长容器启动与健康检查超时 #

TestContainers默认超时可能不足以让LocalStack在高负载配置下完成启动：

localStack.withStartupTimeout(Duration.ofMinutes(5)); // 延长启动超时至5分钟

内容的提问来源于stack exchange，提问作者Scott Lindner