使用AWS SDK v3 lib-storage上传112GB文件致Chrome内存溢出求助

核心原因 #

前端处理112GB这类超大文件时，@aws-sdk/lib-storage默认的并发分片上传会同时将多个文件片段加载到内存中，随着上传推进，未及时释放的内存累积超过Chrome单标签页的内存限制，最终导致崩溃。

具体解决办法 #

1. 调整分片上传的并发数与分片大小 #

降低并发数减少同时驻留内存的分片数量，合理调整分片大小平衡内存占用与上传效率：

const upload = new AWS.Upload({
  client: s3Client,
  params: uploadParams,
  queueSize: 1, // 仅同时上传1个分片，大幅降低内存占用
  partSize: 1024 * 1024 * 10, // 调整为10MB分片（S3要求最小5MB）
  leavePartsOnError: false,
});

注意：partSize不能小于5MB（S3分片上传的最低要求，最后一个分片除外），并发数建议设为1或2，优先保证内存稳定。

2. 优化内存回收与进度回调 #

避免在进度回调中累积不必要的数据，减少内存泄漏风险：

// 简化进度处理，不存储多余历史数据
upload.on("httpUploadProgress", (progress) => {
  console.log(`已上传：${Math.round((progress.loaded / progress.total) * 100)}%`);
  // 手动解除引用，辅助垃圾回收
  progress = null;
});

3. 使用Web Worker隔离上传逻辑 #

将上传逻辑移至Web Worker中，让上传操作在独立线程执行，避免占用主进程内存：

步骤1：创建upload.worker.js

self.onmessage = async (e) => {
  const { s3Config, uploadParams } = e.data;
  // 在Worker中初始化SDK
  const { S3Client } = await import('@aws-sdk/client-s3');
  const { Upload } = await import('@aws-sdk/lib-storage');
  
  const s3Client = new S3Client(s3Config);
  const upload = new Upload({
    client: s3Client,
    params: uploadParams,
    queueSize: 1,
    partSize: 10 * 1024 * 1024,
  });

  upload.on("httpUploadProgress", (progress) => {
    self.postMessage({ type: 'progress', data: progress });
  });

  try {
    await upload.done();
    self.postMessage({ type: 'success' });
  } catch (err) {
    self.postMessage({ type: 'error', data: err.message });
  }
};

步骤2：主页面调用Worker

<script src="bundle.js"></script>
<script>
  const fileInput = document.querySelector('input[type="file"]');
  fileInput.addEventListener('change', async (e) => {
    const file = e.target.files[0];
    const s3Config = {
      region: '...',
      endpoint: '...',
      credentials: {
        accessKeyId: '...',
        secretAccessKey: '...',
        sessionToken: '...',
      }
    };
    const uploadParams = {
      Bucket: '...',
      Key: file.name,
      Body: file
    };

    const uploadWorker = new Worker('upload.worker.js');
    
    uploadWorker.onmessage = (msg) => {
      switch(msg.data.type) {
        case 'progress':
          const progress = msg.data.data;
          console.log(`进度：${Math.round((progress.loaded / progress.total)*100)}%`);
          break;
        case 'success':
          console.log('上传完成');
          uploadWorker.terminate();
          break;
        case 'error':
          console.error('上传失败：', msg.data.data);
          uploadWorker.terminate();
          break;
      }
    };

    uploadWorker.postMessage({ s3Config, uploadParams });
  });
</script>

4. 手动实现分片上传（精细控制内存） #

如果自动分片仍无法满足内存要求，可手动调用S3 Multipart Upload API，逐个读取并上传分片，确保同一时间仅一个分片在内存中：

async function manualMultipartUpload(s3Client, bucket, key, file) {
  const partSize = 10 * 1024 * 1024; // 10MB分片
  const totalParts = Math.ceil(file.size / partSize);
  
  // 初始化分片上传
  const { UploadId } = await s3Client.send(new CreateMultipartUploadCommand({
    Bucket: bucket,
    Key: key,
  }));

  const uploadedParts = [];

  try {
    for (let partNumber = 1; partNumber <= totalParts; partNumber++) {
      const start = (partNumber - 1) * partSize;
      const end = Math.min(start + partSize, file.size);
      const partBlob = file.slice(start, end);

      // 上传单个分片
      const { ETag } = await s3Client.send(new UploadPartCommand({
        Bucket: bucket,
        Key: key,
        UploadId: UploadId,
        PartNumber: partNumber,
        Body: partBlob,
      }));

      uploadedParts.push({ PartNumber: partNumber, ETag });
      console.log(`已上传${Math.round((end / file.size)*100)}%`);
      
      // 释放分片引用，辅助垃圾回收
      partBlob = null;
    }

    // 完成分片上传
    await s3Client.send(new CompleteMultipartUploadCommand({
      Bucket: bucket,
      Key: key,
      UploadId: UploadId,
      MultipartUpload: { Parts: uploadedParts },
    }));
    console.log('上传完成');
  } catch (err) {
    // 出错时中止上传并清理分片
    await s3Client.send(new AbortMultipartUploadCommand({
      Bucket: bucket,
      Key: key,
      UploadId: UploadId,
    }));
    throw err;
  }
}

// 调用示例：manualMultipartUpload(s3Client, 'your-bucket', 'file-key', file);

额外优化建议 #

• 升级@aws-sdk/client-s3和@aws-sdk/lib-storage到最新版本，新版本通常会修复内存泄漏问题；
• 上传期间关闭Chrome其他闲置标签页，释放额外内存；
• 避免在上传同时进行大量DOM渲染、数据计算等内存密集型操作。

内容的提问来源于stack exchange，提问作者ar099968