使用HttpURLConnection跨预签名URL传输文件的并发IO异常解决咨询

当通过预签名GET URL向预签名PUT URL传输文件时，并行调用API超过3次会触发异常。例如调用10次时，前3次能在1分钟内完成，之后Pod的CPU使用率降至0，15分钟后收到如下错误：

java.io.IOException: Error writing request body to server
at java.base/sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3807)
at java.base/sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3789)
...
at java.base/java.lang.Thread.run(Thread.java:833)
Suppressed: java.io.IOException: insufficient data written
    at java.base/sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.close(HttpURLConnection.java:3834)
    at .....
    ... 3 more

当前使用的传输代码：

public ResponseEntity<String> transferData(Config config) throws IOException {

var inputUrl = config.getInputdUrl();
var outputUrl = config.getOutputdUrl();


var uploadConnection = uploader.getUploadConnection(outputUrl);

try (var inputStream = downloader.getDownloadConnection(inputUrl).getInputStream(); 
     var outputStream = uploadConnection.getOutputStream()) {

    IOUtils.copyLarge(inputStream, outputStream);
    outputStream.flush();
    var resp = uploadConnection.getResponseCode();

    return ResponseEntity
            .status(HttpStatus.OK)
            .body("Transfer done with response:" + resp);
} catch (IOException ioe) {
    return ResponseEntity
            .status(HttpStatus.INTERNAL_SERVER_ERROR)
            .body("An  error occurred while transferring data : " + ioe);
}

控制器中调用方式：

new Thread(() -> transferDataToScality(...)).start();
return ResponseEntity.ok("Downloading data in asynchronous mode successfully started");

1. 限制并行线程数，避免资源耗尽 #

无限制创建线程会耗尽系统资源，导致后续请求无法获取连接、CPU或内存，进而出现写入超时或数据不完整。使用Spring的ThreadPoolTaskExecutor自定义线程池控制并行数：

@Bean
public ThreadPoolTaskExecutor transferTaskExecutor() {
    ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
    executor.setCorePoolSize(3); // 核心线程数，根据Pod资源调整
    executor.setMaxPoolSize(5); // 最大线程数
    executor.setQueueCapacity(10); // 任务队列大小
    executor.setThreadNamePrefix("file-transfer-");
    executor.initialize();
    return executor;
}

调用时改用线程池执行：

transferTaskExecutor.execute(() -> transferDataToScality(...));
return ResponseEntity.ok("Downloading data in asynchronous mode successfully started");

2. 为HTTP连接设置超时时间 #

未设置超时会导致线程在网络异常时长期阻塞，占用资源无法释放。在获取HttpURLConnection的方法中添加超时配置：

HttpURLConnection conn = (HttpURLConnection) new URL(url).openConnection();
conn.setConnectTimeout(30000); // 30秒连接超时
conn.setReadTimeout(60000); // 60秒读取超时
// 其他配置...
return conn;

3. 确保连接正确关闭，释放资源 #

即使使用try-with-resources，仍需显式关闭HttpURLConnection，避免连接泄漏：

public ResponseEntity<String> transferData(Config config) {
    HttpURLConnection uploadConnection = null;
    HttpURLConnection downloadConnection = null;
    try {
        var inputUrl = config.getInputdUrl();
        var outputUrl = config.getOutputdUrl();

        uploadConnection = uploader.getUploadConnection(outputUrl);
        downloadConnection = downloader.getDownloadConnection(inputUrl);
        
        try (var inputStream = downloadConnection.getInputStream(); 
             var outputStream = uploadConnection.getOutputStream()) {

            IOUtils.copyLarge(inputStream, outputStream);
            outputStream.flush();
            var resp = uploadConnection.getResponseCode();

            return ResponseEntity
                    .status(HttpStatus.OK)
                    .body("Transfer done with response:" + resp);
        }
    } catch (IOException ioe) {
        return ResponseEntity
                .status(HttpStatus.INTERNAL_SERVER_ERROR)
                .body("An error occurred while transferring data : " + ioe);
    } finally {
        if (uploadConnection != null) uploadConnection.disconnect();
        if (downloadConnection != null) downloadConnection.disconnect();
    }
}

4. 检查预签名URL的有效期 #

预签名URL存在有效期限制，若传输时间超过有效期会被服务端拒绝。确认URL有效期是否覆盖最长传输时间，必要时延长有效期。

5. 调整Pod与JVM资源配置 #

Pod CPU使用率降至0通常是线程全部阻塞导致无法调度。检查Pod的CPU、内存配额，适当调整资源请求与限制；同时优化JVM参数（如-Xmx、-Xms），避免内存不足引发线程阻塞。

内容的提问来源于stack exchange，提问作者ismala