Spring Boot中RestTemplate下载大文件不完整问题求助

环境配置 #

项目使用Spring Boot 2.3.3.RELEASE，父依赖配置如下：

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.3.3.RELEASE</version>
    <relativePath/>
</parent>

问题现象 #

使用RestTemplate从服务器获取文件时，小文件下载正常，但大文件有时无法完整获取。例如一个约12.9M（实际大小13572529字节）的文件，下载后的本地文件大小始终不稳定，但用curl命令可以正确获取该文件。

初始实现代码 #

最初的下载实现代码如下：

@Override
public String downloadFile(String logId, String urlStr, String fName, String localPath) {
    long begin = System.currentTimeMillis();
    String filePath = null;
    try {
        log.info("logId={}-Start to download, fName={}, data={}, filePath={}", logId, fName, urlStr, localPath);

        File dir = new File(localPath);
        if (!dir.exists()) {// Check whether the folder exists
            dir.mkdir();
        }

        filePath = localPath + fName;

        File file = new File(filePath);
        // Do not download the target file if it exists
        if (file.exists()){
            return filePath;
        }

        RequestCallback requestCallback = request -> request.getHeaders()
                .setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));

        // Fetch data as stream instead loading all data into memory
        String finalFilePath = filePath;
        RestTemplate restTemplate = new RestTemplate();
        restTemplate.execute(urlStr, HttpMethod.GET, requestCallback, clientHttpResponse -> {
            
            long contentLength = clientHttpResponse.getHeaders().getContentLength();
            log.info("logId={}-Content-Length in Response Header: {}", logId, contentLength);
            StreamUtils.copy(clientHttpResponse.getBody(), new FileOutputStream(finalFilePath));
            log.info("logId={}-Content-Length in Response Header: {}- file length: {}", logId, contentLength ,file.length());
            return null;
        });

        log.info("logId={}-download success, fName={}, ursStr={}, filePath={}, time cost={}ms", logId, fName, urlStr, filePath, System.currentTimeMillis() - begin);

    } catch (Exception e) {
        log.error("logId={}-download exception, fName={}, urlStr={}, e=", logId, fName, urlStr, e);
        return null;
    }

    log.info("logId={}-download success, fName={}, urlStr={}, filePath={}, time cost={}ms", logId, fName, urlStr, filePath, System.currentTimeMillis() - begin);
    return filePath;
}

调试细节 #

日志显示响应头中的Content-Length始终正确（值为13572529），但本地文件大小不稳定。于是复制StreamUtils.copy()的代码并添加日志跟踪流读取情况：

修改后的代码 #

@Override
public String downloadFile(String logId, String urlStr, String fName, String localPath) {
    long begin = System.currentTimeMillis();
    String filePath = null;
    try {
        log.info("logId={}-Start to download, fName={}, data={}, filePath={}", logId, fName, urlStr, localPath);

        File dir = new File(localPath);
        if (!dir.exists()) {// Check whether the folder exists
            dir.mkdir();
        }

        filePath = localPath + fName;

        File file = new File(filePath);
        // Do not download the target file if it exists
        if (file.exists()){
            return filePath;
        }

        RequestCallback requestCallback = request -> request.getHeaders()
                .setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));

        // Fetch data as stream instead loading all data into memory
        String finalFilePath = filePath;
        RestTemplate restTemplate = new RestTemplate();
        restTemplate.execute(urlStr, HttpMethod.GET, requestCallback, clientHttpResponse -> {
            
            long contentLength = clientHttpResponse.getHeaders().getContentLength();
            log.info("logId={}-Content-Length in Response Header: {}", logId, contentLength);
            copy(clientHttpResponse.getBody(), new FileOutputStream(finalFilePath));
            log.info("logId={}-Content-Length in Response Header: {}- file length: {}", logId, contentLength ,file.length());
            return null;
        });

        log.info("logId={}-download success, fName={}, ursStr={}, filePath={}, time cost={}ms", logId, fName, urlStr, filePath, System.currentTimeMillis() - begin);

    } catch (Exception e) {
        log.error("logId={}-download exception, fName={}, urlStr={}, e=", logId, fName, urlStr, e);
        return null;
    }

    log.info("logId={}-download success, fName={}, urlStr={}, filePath={}, time cost={}ms", logId, fName, urlStr, filePath, System.currentTimeMillis() - begin);
    return filePath;
}


public static int copy(InputStream in, OutputStream out) throws IOException {
    Assert.notNull(in, "No InputStream specified");
    Assert.notNull(out, "No OutputStream specified");
    log.info("inputStream.................|{}", in.available()); // Always changes and less than content length. and less than byteCount
    int byteCount = 0;
    byte[] buffer = new byte[4096]; // 4096 is the value of BUFFER_SIZE in StreamUtils.copy()
    int bytesRead;
    while ((bytesRead = in.read(buffer)) != -1) {
        out.write(buffer, 0, bytesRead);
        byteCount += bytesRead;
    }
    out.flush();
    log.info("inputStream.................|{}-{}", in.available(), byteCount); // in.available() is always 0. byteCount sometimes can get the correct value, `13572529`.
    return byteCount;
}

多次尝试的日志输出 #

inputStream.................|157672
inputStream.................|0-314056
inputStream.................|14320
inputStream.................|0-2206592
inputStream.................|32615
inputStream.................|0-13572529
inputStream.................|14320
inputStream.................|0-546655

日志分析 #

in.available()显示的是clientHttpResponse.getBody()输入流的当前可用字节数，该值始终小于响应头的Content-Length，且最终的byteCount有时能达到正确值13572529，有时却远小于该值。目前对这一现象存在困惑，无法定位问题根源。

提问 #

如何才能通过RestTemplate正确获取完整的大文件？

内容的提问来源于stack exchange，提问作者Nick Dong