如何在Cloud Run托管的Django后台上传超32MB文件？

问题描述 #

需要通过Django管理面板将约800MB的ZIP文件上传至GCP Cloud Storage存储桶，Django API部署在Cloud Run上，但Cloud Run存在32MB的请求大小限制，求可行解决办法。

已尝试方案 #

尝试过使用签名URL（Signed URLs）并将文件分割为30MB的块，但仍收到“413: Request too large”错误。从管理面板上传文件时，仍会发送包含整个文件的请求而非分块上传。以下是本地通过Cloud SQL代理可正常运行的代码：

utils.py #

from django.conf import settings
from google.cloud import storage

def generate_signed_url(file_name, chunk_number):
    storage_client = storage.Client(credentials=settings.GS_CREDENTIALS)
    bucket = storage_client.bucket(settings.GS_BUCKET_NAME)
    blob_name = f'zip_files/{file_name}_chunk_{chunk_number}'
    blob = bucket.blob(blob_name)

    url = blob.generate_signed_url(
        version="v4",
        expiration=3600,
        method="PUT"
    )

    return url

def combine_chunks(bucket_name, blob_name, chunk_count):
    storage_client = storage.Client(credentials=settings.GS_CREDENTIALS)
    bucket = storage_client.bucket(bucket_name)

    # Create a list of the blobs (chunks) to be composed
    blobs = [bucket.blob(f'zip_files/{blob_name}_chunk_{i}') for i in range(chunk_count)]

    # Create a new blob to hold the composed object
    composed_blob = bucket.blob(f'zip_files/{blob_name}')

    # Compose the chunks into a single object
    composed_blob.compose(blobs)

    # Delete the chunks
    for blob in blobs:
        blob.delete()

forms.py #

from django import forms
from django.conf import settings
from .models import Photoset
from .utils import generate_signed_url, combine_chunks
import requests

class PhotosetAdminForm(forms.ModelForm):
    class Meta:
        model = Photoset
        fields = ['name', 'price', "store_photo", "left_photo", "central_photo", "right_photo", "video_length", "photo_count", 'zip_file']

    def save(self, commit=True):
        instance = super().save(commit=False)

        file = self.cleaned_data.get('zip_file')
        if file:
            # Split the file into chunks
            chunk_size = 1024 * 1024 * 30  # 30MB
            chunks = [file.read(chunk_size) for _ in range(0, len(file), chunk_size)]

            # Upload each chunk separately
            for i, chunk in enumerate(chunks):
                signed_url = generate_signed_url(file.name, i)
                headers = {'Content-Type': file.content_type}
                response = requests.put(signed_url, data=chunk, headers=headers)
                if response.status_code != 200:
                    raise ValueError('A chunk could not be uploaded.')
                
            # Combine the chunks into a single file
            combine_chunks(settings.GS_BUCKET_NAME, file.name, len(chunks))

        if commit:
            instance.save()

        return instance

admin.py #

...
@admin.register(Photoset)
class PhotosetAdmin(admin.ModelAdmin):
    list_display = ('id', 'name', 'price', 'insert_date', 'last_modified_date')
    list_display_links = ('name',)
    list_filter = ('models',)
    search_fields = ('name', 'price')
    list_per_page = 25
    form = PhotosetAdminForm
...

zip_file是Photoset模型中的FileField字段。

内容的提问来源于stack exchange，提问作者Nicholas Kemp