Java HttpsUrlConnection setFixedLengthStreamingMode无法恢复S3大文件上传求助

核心逻辑 #

利用AWS S3的**分块上传（Multipart Upload）**特性，把大文件拆分为固定大小的小块（5MB-100MB区间最优，S3要求单块最小5MB，最后一块不限），逐块独立上传。既避免全量加载文件到内存（解决OOM），又能在断网后仅重传未成功的块（实现断点续传）。

具体实现步骤 #

1. 初始化分块上传，获取UploadId #

向S3目标对象路径发送POST请求，携带?uploads参数，拿到唯一的UploadId（后续所有块上传都需用此ID关联），并将其持久化到本地（比如JSON文件）。

URL s3InitUrl = new URL("https://your-bucket.s3.amazonaws.com/target-file?uploads");
HttpsURLConnection conn = (HttpsURLConnection) s3InitUrl.openConnection();
conn.setRequestMethod("POST");
conn.setRequestProperty("Authorization", "AWS your-access-key:calculated-signature");
conn.setRequestProperty("Content-Type", "application/octet-stream");

if (conn.getResponseCode() == 200) {
    // 解析响应XML，提取<UploadId>标签内容
    String uploadId = parseUploadIdFromXml(conn.getInputStream());
    // 保存UploadId到本地文件，用于断点续传
    saveUploadIdToLocal(uploadId);
}
conn.disconnect();

2. 分块读取并上传文件 #

• 按预设块大小逐块读取文件（用FileInputStream分块读取，避免全量加载）
• 每个块发送PUT请求，路径携带?partNumber=X&uploadId=Y参数（X为块序号，从1开始；Y为之前获取的UploadId）
• 块上传成功后，将块序号和响应头中的ETag字段保存到本地，用于后续完成上传或断点续传

File targetFile = new File("your-large-file");
long fileSize = targetFile.length();
int partSize = 10 * 1024 * 1024; // 10MB每块
int partCount = (int) Math.ceil((double) fileSize / partSize);
String uploadId = loadUploadIdFromLocal();

try (FileInputStream fis = new FileInputStream(targetFile)) {
    byte[] buffer = new byte[partSize];
    for (int partNum = 1; partNum <= partCount; partNum++) {
        // 检查本地记录，跳过已上传成功的块
        if (isPartUploaded(partNum)) continue;

        int bytesRead = fis.read(buffer);
        if (bytesRead == -1) break;

        URL partUploadUrl = new URL(String.format(
            "https://your-bucket.s3.amazonaws.com/target-file?partNumber=%d&uploadId=%s",
            partNum, uploadId
        ));
        HttpsURLConnection partConn = (HttpsURLConnection) partUploadUrl.openConnection();
        partConn.setRequestMethod("PUT");
        partConn.setRequestProperty("Authorization", "AWS your-access-key:calculated-signature");
        partConn.setRequestProperty("Content-Length", String.valueOf(bytesRead));
        partConn.setDoOutput(true);

        try (OutputStream os = partConn.getOutputStream()) {
            os.write(buffer, 0, bytesRead);
            os.flush();
        }

        if (partConn.getResponseCode() == 200) {
            String eTag = partConn.getHeaderField("ETag");
            // 保存块序号和ETag到本地
            savePartInfo(partNum, eTag);
        } else {
            // 处理上传失败，可重试3次后标记为待重传
            retryPartUpload(partNum, buffer, bytesRead, uploadId);
        }
        partConn.disconnect();
    }
}

3. 完成分块上传 #

所有块上传完成后，发送POST请求到?uploadId=Y路径，携带包含所有块序号和ETag的XML体，触发S3合并所有块为完整文件。

String uploadId = loadUploadIdFromLocal();
URL completeUrl = new URL("https://your-bucket.s3.amazonaws.com/target-file?uploadId=" + uploadId);
HttpsURLConnection completeConn = (HttpsURLConnection) completeUrl.openConnection();
completeConn.setRequestMethod("POST");
completeConn.setRequestProperty("Authorization", "AWS your-access-key:calculated-signature");
completeConn.setRequestProperty("Content-Type", "application/xml");
completeConn.setDoOutput(true);

// 构建块信息XML
StringBuilder xmlBody = new StringBuilder("<CompleteMultipartUpload>");
List<PartInfo> partInfos = loadAllSavedPartInfos();
for (PartInfo info : partInfos) {
    xmlBody.append("<Part>")
           .append("<PartNumber>").append(info.getPartNum()).append("</PartNumber>")
           .append("<ETag>").append(info.getETag()).append("</ETag>")
           .append("</Part>");
}
xmlBody.append("</CompleteMultipartUpload>");

try (OutputStream os = completeConn.getOutputStream()) {
    os.write(xmlBody.toString().getBytes(StandardCharsets.UTF_8));
    os.flush();
}

if (completeConn.getResponseCode() == 200) {
    // 上传完成，清理本地保存的UploadId和块信息
    cleanUpLocalUploadData();
}
completeConn.disconnect();

4. 断点续传逻辑 #

程序启动时，检查本地是否有未完成的UploadId：

• 若存在，发送GET请求到?uploadId=Y，获取S3端已确认上传成功的块列表，对比本地记录，更新待上传块状态
• 从本地标记的未上传块开始继续上传

String savedUploadId = loadUploadIdFromLocal();
if (savedUploadId != null) {
    URL listPartsUrl = new URL("https://your-bucket.s3.amazonaws.com/target-file?uploadId=" + savedUploadId);
    HttpsURLConnection listConn = (HttpsURLConnection) listPartsUrl.openConnection();
    listConn.setRequestMethod("GET");
    listConn.setRequestProperty("Authorization", "AWS your-access-key:calculated-signature");

    if (listConn.getResponseCode() == 200) {
        // 解析响应XML，获取已上传块列表，更新本地记录
        List<PartInfo> uploadedParts = parseUploadedPartsFromXml(listConn.getInputStream());
        updateLocalPartStatus(uploadedParts);
    }
    listConn.disconnect();
}

关键注意事项 #

• 块大小选择：建议10MB-100MB，平衡请求次数和重试成本；S3要求单块最小5MB，最大5GB，最后一块无大小限制
• 签名计算：需自行实现AWS签名算法（V2或V4），确保Authorization头合法，这是原生实现的核心难点
• 本地存储：用文件（如JSON）或轻量数据库持久化UploadId和块信息，避免程序重启后丢失状态
• 异常处理：针对断网、broken pipe等异常，实现块上传重试逻辑，同时跳过已成功上传的块
• 超时配置：给HttpsURLConnection设置合理的连接超时（setConnectTimeout）和读取超时（setReadTimeout），避免长期阻塞

内容的提问来源于stack exchange，提问作者RAM KUMAR