AWS S3大文件Multipart Upload JavaScript SDK完整实现示例需求

前置要求 #

• AWS账号具备以下S3权限：s3:PutObject、s3:AbortMultipartUpload、s3:ListMultipartUploads、s3:ListParts
• 安装boto3：pip install boto3
• 配置AWS凭证（可通过环境变量AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY、~/.aws/credentials文件或IAM角色）

完整代码实现 #

import boto3
import os
from botocore.exceptions import ClientError

def s3_multipart_upload(file_path, bucket_name, object_key, chunk_size=100*1024*1024):
    """
    超大文件S3分片上传实现
    :param file_path: 本地文件路径
    :param bucket_name: S3存储桶名称
    :param object_key: S3对象键（上传后的文件名）
    :param chunk_size: 分片大小，默认100MB（S3要求最小分片5MB，最后一片无限制）
    """
    s3_client = boto3.client('s3')
    file_size = os.path.getsize(file_path)
    
    # 小文件直接用普通上传
    if file_size <= chunk_size:
        try:
            s3_client.upload_file(file_path, bucket_name, object_key)
            print(f"文件 {file_path} 已通过普通上传完成，存储为 {bucket_name}/{object_key}")
            return
        except ClientError as e:
            print(f"普通上传失败: {e}")
            raise
    
    # 初始化分片上传
    try:
        response = s3_client.create_multipart_upload(Bucket=bucket_name, Key=object_key)
        upload_id = response['UploadId']
        print(f"已初始化分片上传，Upload ID: {upload_id}")
    except ClientError as e:
        print(f"初始化分片上传失败: {e}")
        raise
    
    parts = []
    part_number = 1
    uploaded_bytes = 0
    
    try:
        with open(file_path, 'rb') as f:
            while uploaded_bytes < file_size:
                chunk = f.read(chunk_size)
                if not chunk:
                    break
                
                # 上传当前分片
                try:
                    response = s3_client.upload_part(
                        Bucket=bucket_name,
                        Key=object_key,
                        PartNumber=part_number,
                        UploadId=upload_id,
                        Body=chunk
                    )
                    parts.append({
                        'PartNumber': part_number,
                        'ETag': response['ETag']
                    })
                    print(f"已上传分片 {part_number}，已上传字节数: {uploaded_bytes + len(chunk)}/{file_size}")
                    uploaded_bytes += len(chunk)
                    part_number += 1
                except ClientError as e:
                    print(f"上传分片 {part_number} 失败: {e}")
                    raise
        
        # 完成分片上传
        s3_client.complete_multipart_upload(
            Bucket=bucket_name,
            Key=object_key,
            UploadId=upload_id,
            MultipartUpload={'Parts': parts}
        )
        print(f"分片上传完成，文件 {file_path} 已存储为 {bucket_name}/{object_key}")
    
    except Exception as e:
        # 出错时终止分片上传，避免残留未完成任务
        print(f"上传过程出错，正在终止分片上传: {e}")
        try:
            s3_client.abort_multipart_upload(
                Bucket=bucket_name,
                Key=object_key,
                UploadId=upload_id
            )
            print("分片上传已终止")
        except ClientError as abort_err:
            print(f"终止分片上传失败: {abort_err}")
        raise

# 示例调用
if __name__ == "__main__":
    FILE_PATH = "/path/to/your/large/file"
    BUCKET_NAME = "your-s3-bucket-name"
    OBJECT_KEY = "uploaded-large-file.ext"
    CHUNK_SIZE = 100 * 1024 * 1024  # 100MB
    
    s3_multipart_upload(FILE_PATH, BUCKET_NAME, OBJECT_KEY, CHUNK_SIZE)

代码关键说明 #

• 分片大小选择：默认100MB，可根据网络带宽调整（建议范围100MB-1GB），S3要求除最后一片外，其他分片最小为5MB
• 普通上传降级：小文件直接用upload_file，无需分片，提升小文件上传效率
• 异常兜底：任何步骤出错都会自动终止分片上传，避免S3中残留未完成的上传任务（这类任务会占用存储并产生额外费用）
• 进度监控：实时打印已上传分片和字节数，方便追踪上传状态

扩展建议 #

• 断点续传：将upload_id和已上传分片信息保存到本地文件，下次启动时先调用list_parts查询已上传分片，跳过重复上传
• 重试机制：通过boto3配置重试次数，比如：boto3.client('s3', config=Config(retries={'max_attempts': 5}))
• 文件校验：上传完成后调用head_object获取文件ETag，与本地文件的分片MD5组合值对比，确保上传完整性
• 并行上传：使用线程池并行上传多个分片，进一步提升大文件上传速度（注意控制线程数，避免超出S3并发限制）

内容的提问来源于stack exchange，提问作者Jim Chertkov