Docker部署Django大文件上传触发Gunicorn Worker超时OOM问题求助

我将Django应用Docker化后，用户可直接上传文件至DigitalOcean Spaces。小文件在本地及Heroku部署环境中上传正常，但上传200MB以上大文件时，会触发以下错误：

[2024-09-29 19:00:51 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:7)
web-1     | [2024-09-29 19:00:52 +0000] [1] [ERROR] Worker (pid:7) was sent SIGKILL! Perhaps out of memory?
web-1     | [2024-09-29 19:00:52 +0000] [29] [INFO] Booting worker with pid: 29

错误约在上传发起30秒后出现，推测是Gunicorn因未收到响应触发超时。目前除了不推荐的延长超时时间外，不知该如何解决。以下是相关代码：

views.py #

@csrf_protect
def transcribe_submit(request):
    if request.method == 'POST':
        form = UploadFileForm(request.POST, request.FILES)
        if form.is_valid():
            uploaded_file = request.FILES['file']

            request.session['uploaded_file_name'] = uploaded_file.name
            request.session['uploaded_file_size'] = uploaded_file.size

            session_id = str(uuid.uuid4())
            request.session['session_id'] = session_id

            try:
                transcribed_doc, created = TranscribedDocument.objects.get_or_create(id=session_id)
                transcribed_doc.audio_file = uploaded_file
                transcribed_doc.save()
          ...

            except Exception as e:
                # Log the error and respond with a server error status
                print(f"Error occurred: {str(e)}")
                return HttpResponse(status=500)

          ...
        else:
            return HttpResponse(status=500)
    else:
        form = UploadFileForm()
        
    return render(request, 'transcribe/transcribe-en.html', {"form": form})

forms.py #

def validate_audio_language(value):
    #code to validate audio language
    if value not in allowed_languages:
        raise ValidationError("Error")

def validate_output_file_type(value):
    #code to validate file type
    if value not in output_file_type:
        raise ValidationError("Error")
    
class UploadFileForm(forms.Form):
    file = forms.FileField(validators=[validate_file])

docker-compose.yml #

#version: "3.9"
services:
  web:
    build: .
    #command: python /code/manage.py runserver  0.0.0.0:8000
    command: gunicorn mysite.wsgi -b 0.0.0.0:8000 --reload
    volumes:
      - .:/code
    ports:
      - 8000:8000
    depends_on:
      - db
      - redis
      - celery
    environment:
      -  "DJANGO_SECRET_KEY="
      
    user: user-me
  db:
    image: postgres:13
    volumes:
      - postgres_data:/var/lib/postgresql/data/
    environment:
      - "POSTGRES_HOST_AUTH_METHOD=trust"
  redis:
    image: redis:6
    ports:
      - 6379:6379
  celery:
    build: .
    command: celery -A mysite worker --loglevel=info 
    volumes:
      - .:/code
    depends_on:
      - redis
      - db
    environment:
      -  "DJANGO_SECRET_KEY="

    user: user-me
volumes:
  postgres_data:

解决方案 #

1. 直接客户端上传到DigitalOcean Spaces（推荐）

跳过Django后端中转大文件，让浏览器直接上传至Spaces：

• 后端生成带签名的临时上传URL（用boto3或django-storages的API），返回给前端
• 前端使用该URL直接将文件上传到Spaces
• 文件上传完成后，前端通知后端文件在Spaces中的路径，后端仅需记录该路径即可

此方案彻底避免后端处理大文件的内存占用和超时问题，流程更高效。

2. 流式上传到Spaces，避免加载整个文件到内存

若必须通过后端中转，修改代码让Django将文件流直接传输到Spaces，而非将整个文件读入内存：

# 示例代码片段
import boto3
from botocore.config import Config

def upload_file_to_spaces(file_stream, key):
    s3 = boto3.client('s3',
                      endpoint_url='https://nyc3.digitaloceanspaces.com',
                      aws_access_key_id='YOUR_ACCESS_KEY',
                      aws_secret_access_key='YOUR_SECRET_KEY',
                      config=Config(signature_version='s3v4'))
    s3.upload_fileobj(file_stream, 'your-space-name', key)

# 在视图中替换原保存逻辑
uploaded_file = request.FILES['file']
file_key = f"uploads/{session_id}/{uploaded_file.name}"
upload_file_to_spaces(uploaded_file, file_key)
# 仅保存file_key到TranscribedDocument，而非整个文件
transcribed_doc.audio_file = file_key  # 假设字段为CharField存储路径
transcribed_doc.save()

3. 优化Gunicorn配置（临时缓解，不推荐作为最终方案）

若以上方案暂时无法实施，可调整Gunicorn的超时和worker配置：

• 在docker-compose的web命令中添加超时参数：gunicorn mysite.wsgi -b 0.0.0.0:8000 --reload --timeout 120（延长超时至120秒）
• 增加worker内存限制，或使用异步worker（如gevent），但仅为治标之法。

4. 配置Django文件上传设置

确保Django开启流式上传，避免内存过载：

# settings.py
FILE_UPLOAD_MAX_MEMORY_SIZE = 10485760  # 10MB，超过则写入临时文件
FILE_UPLOAD_TEMP_DIR = '/tmp'  # 设置临时文件存储目录

内容的提问来源于stack exchange，提问作者tthheemmaannii