ECS中Django与Celery服务如何共享Docker卷实现异步S3上传？

你遇到的核心问题是：ECS Fargate中独立任务的默认空卷（empty volume）是任务级别的临时存储，Django服务所在的rest-prod-task和Celery任务所在的独立任务无法共享该卷。以下是无需EFS的可行方案：

1. 将Django与Celery容器部署在同一ECS任务中 #

Fargate中，同一任务内的容器可以共享网络、环境变量和存储卷。调整任务定义，把Django服务和Celery Worker容器放在同一个任务家族下：

{
  "family": "django-celery-prod-task",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "2048",
  "memory": "4096",
  "containerDefinitions": [
    {
      "name": "rest-container",
      "image": "docker-image-url",
      "mountPoints": [
        {
          "containerPath": "/shared",
          "sourceVolume": "shared-volume",
          "readOnly": false
        }
      ],
      "environment": [
        {"name": "MEDIA_ROOT", "value": "/shared"},
        {"name": "DJANGO_SETTINGS_MODULE", "value": "your_project.settings.prod"}
      ],
      "portMappings": [{"containerPort": 8000, "hostPort": 8000}],
      "essential": true
    },
    {
      "name": "celery-worker-container",
      "image": "docker-image-url",
      "mountPoints": [
        {
          "containerPath": "/shared",
          "sourceVolume": "shared-volume",
          "readOnly": false
        }
      ],
      "environment": [
        {"name": "MEDIA_ROOT", "value": "/shared"},
        {"name": "DJANGO_SETTINGS_MODULE", "value": "your_project.settings.prod"},
        {"name": "CELERY_BROKER_URL", "value": "redis://your-redis-endpoint:6379/0"}
      ],
      "command": ["celery", "-A", "your_project", "worker", "--loglevel=info"],
      "essential": true
    }
  ],
  "volumes": [
    {
      "name": "shared-volume"
    }
  ]
}

2. 修改Django文件上传路径到共享卷 #

调整模型的文件上传路径生成函数，让文件直接保存到共享卷目录：

import os
import uuid
from django.db import models

def generate_file_upload_path(instance: "MyModel", filename: str):
    _, ext = os.path.splitext(filename)
    randomised_hex = uuid.uuid4().hex
    # 直接保存到共享卷路径
    return f"/shared/{instance.id}-{randomised_hex}{ext}"

class MyModel(models.Model):
    media_file = models.FileField(upload_to=generate_file_upload_path, blank=True, null=True)
    thumbnail_file = models.FileField(upload_to=generate_file_upload_path, blank=True, null=True)

3. 调整Celery任务读取共享卷文件并上传S3 #

修改Celery任务代码，直接从共享卷读取文件，上传完成后可删除本地临时文件：

import os
from celery import shared_task
from django.conf import settings
from .models import MyModel
import boto3

@shared_task
def process_media_file(task_data):
    media_meta_id = task_data["media_meta_id"]
    try:
        obj = MyModel.objects.get(id=media_meta_id)
        # 从共享卷读取文件路径
        media_local_path = obj.media_file.path
        thumbnail_local_path = obj.thumbnail_file.path

        # 初始化S3客户端
        s3_client = boto3.client("s3")
        bucket_name = settings.AWS_STORAGE_BUCKET_NAME

        # 上传媒体文件到S3
        media_s3_key = f"media/{os.path.basename(media_local_path)}"
        s3_client.upload_file(media_local_path, bucket_name, media_s3_key)
        # 更新模型字段为S3路径（可选，根据你的存储策略调整）
        obj.media_file = media_s3_key

        # 上传缩略图到S3
        thumbnail_s3_key = f"thumbnails/{os.path.basename(thumbnail_local_path)}"
        s3_client.upload_file(thumbnail_local_path, bucket_name, thumbnail_s3_key)
        obj.thumbnail_file = thumbnail_s3_key

        obj.save()

        # 删除共享卷中的临时文件
        if os.path.exists(media_local_path):
            os.remove(media_local_path)
        if os.path.exists(thumbnail_local_path):
            os.remove(thumbnail_local_path)
    except MyModel.DoesNotExist:
        return "Media record not found"
    except Exception as e:
        # 错误处理逻辑
        print(f"Upload failed: {str(e)}")
        raise

4. 解决容器权限问题 #

确保容器对/shared目录有读写权限，在Dockerfile中添加以下命令：

# 创建共享目录并修改权限（假设运行用户为django）
RUN mkdir -p /shared && chown -R django:django /shared
USER django

关键注意事项 #

• Fargate的空卷是临时存储，任务停止或重启后数据会丢失，因此必须在Celery任务完成上传后删除本地文件。
• 确保Django和Celery容器使用相同的Docker镜像或一致的环境变量配置（如MEDIA_ROOT）。
• 若需要多实例部署，此方案仅适用于单任务实例，多实例情况下仍需依赖EFS等共享存储（你暂不考虑的话可忽略）。

内容的提问来源于stack exchange，提问作者abinash_123_