如何使用boto3和Lambda实现HTML页面上传大于10MB文件到S3存储桶

原有方案的核心瓶颈是AWS API Gateway的Payload硬限制为10MB，同时Lambda也有运行时长、临时存储限制，无法处理大文件上传，必须更换架构，不走「文件上传到API Gateway再转发给Lambda再传到S3」的链路。
采用预签名URL + S3分片上传的组合方案，优势如下：

• 文件直接从前端上传到S3，不经过API Gateway和Lambda，无10MB大小限制
• 预签名URL由Lambda生成，保留权限管控能力，不会公开S3桶的写权限
• 分片上传支持断点续传，大文件上传失败不需要重传整个文件
• 自动适配小文件：小于10MB的文件可以直接用单文件预签名PUT上传，逻辑更简单

1. 后端Lambda修改（负责生成预签名URL） #

Lambda的职责从处理文件上传改为：接收前端传来的文件名、文件大小、分片信息，生成对应的预签名URL返回给前端，同时优化文件后缀判断逻辑，适配所有后缀场景：

import json
import boto3
import os
from create_bucket import create_bucket

s3 = boto3.client('s3')
# 预签名URL有效期，单位秒
EXPIRATION = 3600
# 分片大小阈值，小于该值走单文件上传，大于等于走分片，这里设为10MB
PART_SIZE_THRESHOLD = 10 * 1024 * 1024
bucket_without_extension = "filetypewithoutextension"

def lambda_handler(event, context):
    # 解析前端请求参数
    req = json.loads(event['body'])
    file_name = req['file_name']
    file_size = req['file_size']
    upload_type = req.get('upload_type', 'single') # single单文件，multipart分片
    
    # 处理文件后缀，适配所有后缀场景
    ext = os.path.splitext(file_name)[-1].lower().lstrip('.')
    if not ext:
        bucket_name = bucket_without_extension
    else:
        bucket_name = f"thisismycomplicatedbucketfiletype-{ext}"
    
    # 检查桶是否存在，不存在则创建
    try:
        s3.head_bucket(Bucket=bucket_name)
    except s3.exceptions.ClientError:
        create_bucket(bucket_name)
    
    # 单文件上传预签名URL生成
    if upload_type == 'single':
        presigned_url = s3.generate_presigned_url(
            ClientMethod='put_object',
            Params={
                'Bucket': bucket_name,
                'Key': file_name,
                'ServerSideEncryption': 'aws:kms'
            },
            ExpiresIn=EXPIRATION
        )
        return {
            'statusCode': 200,
            'headers': {'Access-Control-Allow-Origin': '*'},
            'body': json.dumps({'presigned_url': presigned_url, 'bucket': bucket_name, 'key': file_name})
        }
    
    # 分片上传场景
    elif upload_type == 'multipart':
        # 初始化分片上传
        upload_id = s3.create_multipart_upload(
            Bucket=bucket_name,
            Key=file_name,
            ServerSideEncryption='aws:kms'
        )['UploadId']
        # 计算分片数量
        part_count = (file_size + PART_SIZE_THRESHOLD - 1) // PART_SIZE_THRESHOLD
        # 生成每个分片的预签名URL
        presigned_urls = []
        for part_num in range(1, part_count + 1):
            url = s3.generate_presigned_url(
                ClientMethod='upload_part',
                Params={
                    'Bucket': bucket_name,
                    'Key': file_name,
                    'UploadId': upload_id,
                    'PartNumber': part_num
                },
                ExpiresIn=EXPIRATION
            )
            presigned_urls.append({'part_number': part_num, 'url': url})
        
        return {
            'statusCode': 200,
            'headers': {'Access-Control-Allow-Origin': '*'},
            'body': json.dumps({
                'upload_id': upload_id,
                'presigned_urls': presigned_urls,
                'part_size': PART_SIZE_THRESHOLD,
                'bucket': bucket_name,
                'key': file_name
            })
        }
    
    # 完成分片上传的接口
    elif upload_type == 'complete_multipart':
        upload_id = req['upload_id']
        parts = req['parts']
        s3.complete_multipart_upload(
            Bucket=bucket_name,
            Key=file_name,
            UploadId=upload_id,
            MultipartUpload={'Parts': parts}
        )
        return {
            'statusCode': 200,
            'headers': {'Access-Control-Allow-Origin': '*'},
            'body': json.dumps({'message': 'File uploaded successfully!'})
        }

2. 前端HTML修改（增加JS上传逻辑） #

前端不再直接提交Form到API Gateway，而是先请求预签名URL，再直接上传到S3：

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Upload File to S3 Bucket</title>
</head>
<body>
    <h3>Upload File to S3 Bucket</h3>
    <div>
        <h1>Select File:</h1>
        <input type="file" id="fileInput" name="file" />
        <button id="uploadBtn">Upload</button>
        <div id="progress"></div>
    </div>

    <script>
        const API_ENDPOINT = "https://4oss3kkck9.execute-api.us-east-1.amazonaws.com/api/uploadhtml";
        const PART_SIZE = 10 * 1024 * 1024; // 10MB分片阈值

        document.getElementById('uploadBtn').addEventListener('click', async () => {
            const file = document.getElementById('fileInput').files[0];
            if (!file) {
                alert('Please select a file first');
                return;
            }
            const progressDiv = document.getElementById('progress');
            progressDiv.innerText = 'Preparing upload...';

            // 小文件直接单文件上传
            if (file.size < PART_SIZE) {
                // 请求单文件预签名URL
                const presignRes = await fetch(API_ENDPOINT, {
                    method: 'POST',
                    headers: {'Content-Type': 'application/json'},
                    body: JSON.stringify({
                        file_name: file.name,
                        file_size: file.size,
                        upload_type: 'single'
                    })
                });
                const {presigned_url} = await presignRes.json();
                // 直接上传到S3
                progressDiv.innerText = 'Uploading...';
                await fetch(presigned_url, {
                    method: 'PUT',
                    body: file
                });
                progressDiv.innerText = 'File uploaded successfully!';
                return;
            }

            // 大文件走分片上传
            progressDiv.innerText = 'Generating multipart upload URLs...';
            // 请求分片预签名URL
            const presignRes = await fetch(API_ENDPOINT, {
                method: 'POST',
                headers: {'Content-Type': 'application/json'},
                body: JSON.stringify({
                    file_name: file.name,
                    file_size: file.size,
                    upload_type: 'multipart'
                })
            });
            const {upload_id, presigned_urls, part_size, bucket, key} = await presignRes.json();
            const parts = [];
            let uploaded = 0;

            // 并发上传每个分片
            for (let i = 0; i < presigned_urls.length; i++) {
                const {part_number, url} = presigned_urls[i];
                const start = (part_number - 1) * part_size;
                const end = Math.min(start + part_size, file.size);
                const blob = file.slice(start, end);
                
                const res = await fetch(url, {method: 'PUT', body: blob});
                const etag = res.headers.get('ETag');
                parts.push({PartNumber: part_number, ETag: etag});
                
                uploaded += blob.size;
                progressDiv.innerText = `Uploading: ${Math.round(uploaded / file.size * 100)}%`;
            }

            // 通知后端完成分片上传
            progressDiv.innerText = 'Completing upload...';
            await fetch(API_ENDPOINT, {
                method: 'POST',
                headers: {'Content-Type': 'application/json'},
                body: JSON.stringify({
                    file_name: file.name,
                    upload_type: 'complete_multipart',
                    upload_id: upload_id,
                    parts: parts
                })
            });
            progressDiv.innerText = 'File uploaded successfully!';
        });
    </script>
</body>
</html>

3. 必要配置调整 #

• API Gateway需要开启CORS配置，允许前端跨域请求，同时把Lambda的集成请求改为透传JSON body，不需要处理multipart/form-data
• 给Lambda的执行角色增加S3权限：s3:PutObject、s3:CreateMultipartUpload、s3:UploadPart、s3:CompleteMultipartUpload、s3:HeadBucket
• S3桶需要配置CORS规则，允许前端的PUT请求，示例CORS规则：

<CORSConfiguration>
 <CORSRule>
   <AllowedOrigin>*</AllowedOrigin> <!-- 建议替换为你的前端域名 -->
   <AllowedMethod>PUT</AllowedMethod>
   <AllowedMethod>POST</AllowedMethod>
   <AllowedHeader>*</AllowedHeader>
   <ExposeHeader>ETag</ExposeHeader>
 </CORSRule>
</CORSConfiguration>

内容的提问来源于stack exchange，提问作者Васко Бошков