多线程批量上传文件至AWS S3时抛出ClosedChannelException异常

DevOps 小满

2026-5-28

解决多线程批量上传AWS S3时的ClosedChannelException异常

你用ExecutorService开100个线程上传S3时碰到的ClosedChannelException，我之前做批量上传的时候也遇到过，本质上是文件通道被意外关闭导致的，咱们来一步步分析解决：

先把你遇到的异常贴出来方便参考：

com.amazonaws.SdkClientException: Unable to execute HTTP request: null | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1114) ~[aws-java-sdk-core-1.11.248.jar!/:na] | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1064) ~[aws-java-sdk-core-1.11.248.jar!/:na] | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) ~[aws-java-sdk-core-1.11.248.jar!/:na] | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[aws-java-sdk-core-1.11.248.jar!/:na] | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[aws-java-sdk-core-1.11.248.jar!/:na] | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[aws-java-sdk-core-1.11.248.jar!/:na] | at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[aws-java-sdk-core-1.11.248.jar!/:na] | at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[aws-java-sdk-core-1.11.248.jar!/:na] | at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4325) ~[aws-java-sdk-s3-1.11.248.jar!/:na] | at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4272) ~[aws-java-sdk-s3-1.11.248.jar!/:na] | at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1749) ~[aws-java-sdk-s3-1.11.248.jar!/:na] | ... 9 common frames omitted | Caused by: java.nio.channels.ClosedChannelException: null | at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110) ~[na:1.8.0_171] | at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:147) ~[na:1.8.0_171] | at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65) ~[na:1.8.0_171] | at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[na:1.8.0_171] | at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[na:1.8.0_171] | at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) ~[aws-java-sdk-core-1.11.248.jar!/:na] | at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) ~[aws-java-sdk-core-1.11.248.jar!/:na] | at com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:107) ~[aws-java-sdk-core-1.11.248.jar!/:na] | at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) ~[aws-java-sdk-core-1.11.248.jar!/:na] | at com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream.read(MD5DigestCalculatingInputStream.java:128) ~[aws-java-sdk-s3-1.11.248.jar!/:na] | at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[na:1.8.0_171] | at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[na:1.8.0_171] | at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[na:1.8.0_171] | at com.amazonaws.internal.SdkBufferedInputStream.read(SdkBufferedInputStream.java:76) ~[aws-java-sdk-core-1.11.248.jar!/:na] | at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) ~[aws-java-sdk-core-1.11.248.jar!/:na] | at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) ~[aws-java-sdk-core-1.11.248.jar!/:na] | at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) ~[aws-java-sdk-core-1.11.248.jar!/:na] | at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:140) ~[httpcore-4.4.9.jar!/:4.4.9] | at com.amazonaws.http.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:160) ~[aws-java-sdk-core-1.11.248.jar!/:na] | at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) ~[httpcore-4.4.9.jar!/:4.4.9] | at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:160) ~[httpclient-4.5.4.jar!/:4.5.4] | at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) ~[httpcore-4.4.9.jar!/:4.4.9] | at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doSendRequest(SdkHttpRequestExecutor.java:63) ~[aws-java-sdk-core-1.11.248.jar!/:na] | at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) ~[httpcore-4.4.9.jar!/:4.4.9] | at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.4.jar!/:4.5.4] | at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[httpclient-4.5.4.jar!/:4.5.4] | at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.4.jar!/:4.5.4] | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.4.jar!/:4.5.4] | at org.apache.ht

可能的原因分析

共享文件流/通道：如果多个线程复用了同一个FileInputStream或者文件通道，当其中一个线程关闭流后，其他线程再读写就会触发这个异常。
线程数超过连接池限制：AWS SDK底层的Apache HttpClient默认连接池大小是50左右，你开100个线程的话，大量请求会竞争连接，导致连接被强制关闭或者资源耗尽。
文件被外部进程占用：上传过程中如果文件被其他程序修改或锁定，也会导致通道意外关闭。

具体解决方案

1. 每个线程使用独立的文件流

绝对不要在多个线程之间共享文件输入流，每个上传任务都应该单独打开流，并用try-with-resources自动关闭，示例代码如下：

ExecutorService executor = Executors.newFixedThreadPool(20); // 先调整线程数到合理范围
List<String> filePaths = Arrays.asList("file1.txt", "file2.txt", "...");

for (String path : filePaths) {
    executor.submit(() -> {
        // 每个任务单独打开流，try-with-resources自动关闭
        try (FileInputStream fis = new FileInputStream(new File(path))) {
            PutObjectRequest request = new PutObjectRequest("your-bucket-name", "remote-key", fis, new ObjectMetadata());
            amazonS3.putObject(request);
        } catch (IOException | AmazonServiceException e) {
            // 处理异常
            e.printStackTrace();
        }
    });
}

executor.shutdown();
executor.awaitTermination(1, TimeUnit.HOURS);

2. 合理配置线程池大小

不要盲目开100个线程，建议先调整到20-50之间（和AWS SDK的默认连接池匹配），如果机器性能足够，可以后续再逐步调优，避免资源竞争。

3. 自定义AWS SDK的连接池配置

你可以通过ClientConfiguration来增大连接池的最大连接数，同时调整超时参数，避免连接被过早关闭：

ClientConfiguration clientConfig = new ClientConfiguration();
clientConfig.setMaxConnections(100); // 匹配你的线程数
clientConfig.setSocketTimeout(30000); // 30秒超时
clientConfig.setConnectionTimeout(5000); // 5秒连接超时

AmazonS3 amazonS3 = AmazonS3ClientBuilder.standard()
        .withClientConfiguration(clientConfig)
        .withCredentials(new DefaultAWSCredentialsProviderChain())
        .withRegion(Regions.US_EAST_1) // 替换成你的区域
        .build();

4. 确保文件的独占访问

上传前检查文件是否被其他进程占用，比如可以尝试获取文件的独占锁，或者确保上传期间没有其他程序修改该文件。

内容的提问来源于stack exchange，提问作者Vikram Singh Shekhawat