解决AWS S3文件打包Zip上传后损坏问题及性能优化

核心问题分析 #

• 错误的流操作：你直接向pipedOutputStream写入数据，绕过了Zip4J的ZipOutputStream，导致Zip文件的结构（比如条目头、校验信息）没有被正确写入，这是文件损坏的根本原因。
• PipedStream缓冲区溢出风险：默认PipedStream的缓冲区只有1024字节，多线程下如果写入速度远快于读取，会导致线程阻塞甚至数据丢失。
• 未正确完成Zip流收尾：缺少finish()操作，Zip文件的中央目录等关键结构未被写入，导致文件无法被正常解析。

修复后的代码实现 #

1. 修正下载打包线程 #

private Thread getDownloadAndZipThread(List<String> keys, ZipParameters parameters) {
    return new Thread(() -> {
        long start = System.currentTimeMillis();
        try (final ZipOutputStream zipOutputStream = new ZipOutputStream(pipedOutputStream)) {
            zipOutputStream.setLevel(CompressionLevel.FASTEST.getLevel());
            for (String key : keys) {
                try {
                    parameters.setFileNameInZip(key);
                    zipOutputStream.putNextEntry(parameters);
                    // 必须通过ZipOutputStream写入数据，保证Zip结构完整
                    byte[] fileBytes = s3Service.getFile(key, BUCKET_NAME).readAllBytes();
                    zipOutputStream.write(fileBytes);
                    zipOutputStream.closeEntry();
                } catch (IOException e) {
                    throw new RuntimeException("处理文件失败: " + key, e);
                }
            }
            // 主动完成Zip流，写入中央目录等收尾结构
            zipOutputStream.finish();
        } catch (Exception e) {
            logger.log("ZipService - getDownloadAndZipThread - error: " + e);
            Thread.currentThread().interrupt();
        }
        long executedTime = System.currentTimeMillis() - start;
        logger.log("ZipService - getDownloadAndZipThread - execution time: " + executedTime);
    });
}

2. 修正上传线程 #

private Thread getS3Out(String filename) {
    return new Thread(() -> {
        long start = System.currentTimeMillis();
        try {
            s3Service.multipartUpload(filename, BUCKET_NAME, pipedInputStream);
        } catch (final Exception all) {
            logger.log("Failed to process outputStream due to error: " + all);
            Thread.currentThread().interrupt();
        } finally {
            try {
                pipedInputStream.close();
            } catch (IOException e) {
                logger.log("关闭输入流失败: " + e);
            }
        }
        long executedTime = System.currentTimeMillis() - start;
        logger.log("ZipService - getS3Out - execution time: " + executedTime);
    });
}

3. 线程调用修正 #

String filename = generateZipName();
ZipParameters parameters = new ZipParameters();

// 初始化大缓冲区Piped流，避免阻塞（设置为64KB）
pipedOutputStream = new PipedOutputStream();
pipedInputStream = new PipedInputStream(pipedOutputStream, 65536);

final Thread downloadAndZip = getDownloadAndZipThread(keys, parameters); 
final Thread upload = getS3Out(filename);

downloadAndZip.start();
upload.start();
try {
    downloadAndZip.join();
    upload.join();
} catch (InterruptedException e) {
    logger.log("ZipService - 线程等待中断: " + e);
    downloadAndZip.interrupt();
    upload.interrupt();
    throw new RuntimeException(e);
}

Lambda环境适配优化 #

• 内存配置：Lambda内存越高，CPU和网络带宽也越高，建议配置至少2048MB内存以提升大文件处理效率。
• 临时存储：若处理10GB级文件，可开启Lambda临时存储扩容（最大10GB），替代PipedStream做本地中转，降低内存压力。
• 分段上传优化：将S3分段上传的块大小设置为8MB或16MB，适配Lambda内存限制，避免OOM。

内容的提问来源于stack exchange，提问作者Liubomur Pryimak