Django+S3+Zappa部署后文件上传报HeadObject 400请求错误

问题背景 #

基于Django和S3（已启用私有、公共ACL）的项目通过Zappa部署后，在Django管理后台向指定S3存储桶上传文件时，触发HeadObject操作的400 Bad Request错误。

错误详情 #

ClientError at /admin/main/document/1/change/
An error occurred (400) when calling the HeadObject operation: Bad Request
Request Method: POST
Request URL:    https://xxxxx.execute-api.us-east-1.amazonaws.com/prd/admin/main/document/1/change/
Django Version: 3.2.3
Exception Type: ClientError
Exception Value:    
An error occurred (400) when calling the HeadObject operation: Bad Request
Exception Location: /var/runtime/botocore/client.py, line 719, in _make_api_call
Python Executable:  /var/lang/bin/python3.8
Python Version: 3.8.13
Python Path:    
['/var/task',
 '/opt/python/lib/python3.8/site-packages',
 '/opt/python',
 '/var/runtime',
 '/var/lang/lib/python38.zip',
 '/var/lang/lib/python3.8',
 '/var/lang/lib/python3.8/lib-dynload',
 '/var/lang/lib/python3.8/site-packages',
 '/opt/python/lib/python3.8/site-packages',
 '/var/task',
 '/var/task/odf',
 '/var/task/odf',
 '/var/task/odf',
 '/var/task/odf',
 '/var/task/odf',
 '/var/task/odf',
 '/var/task/odf']
Server time:    Wed, 17 Aug 2022 05:20:25 +0000

/var/task/storages/backends/s3boto3.py, line 469, in exists
            self.connection.meta.client.head_object(Bucket=self.bucket_name, Key=name) 

相关配置 #

zappa_settings.json #

{
    "prd": {
        "aws_region": "us-east-1",
        "django_settings": "xxx.settings",
        "profile_name": "default",
        "project_name": "xxxx",
        "runtime": "python3.8",
        "s3_bucket": "zappa-xxx-prd",
        "environment_variables": {
            "AWS_DEFAULT_REGION": "us-east-1",
            "AWS_S3_ACCESS_KEY_ID": "xxxxxxx",
            "AWS_S3_SECRET_ACCESS_KEY": "xxxxxx",
            "AWS_ACCESS_KEY_ID": "xxxxx",
            "AWS_SECRET_ACCESS_KEY": "xxxxxx"
        }
    }
}

用户AWS通用策略 #

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "lambda:*",
                "s3:*",
                "events:*",
                "iam:CreateServiceSpecificCredential",
                "iam:GetRole",
                "iam:CreateRole",
                "iam:PutRolePolicy",
                "iam:PassRole",
                "iam:CreateServiceLinkedRole",
                "apigateway:PUT",
                "apigateway:DELETE",
                "apigateway:PATCH",
                "apigateway:POST",
                "apigateway:GET",
                "logs:DescribeLogStreams",
                "logs:FilterLogEvents",
                "cloudformation:DescribeStackResource",
                "cloudformation:DescribeStacks",
                "cloudformation:CreateStack",
                "cloudformation:DeleteStack",
                "cloudformation:UpdateStack",
                "cloudformation:ListStackResources"
            ],
            "Resource": "*"
        }
    ]
}

Zappa Lambda执行角色默认策略 #

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:*"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "lambda:InvokeFunction"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "xray:PutTraceSegments",
                "xray:PutTelemetryRecords"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AttachNetworkInterface",
                "ec2:CreateNetworkInterface",
                "ec2:DeleteNetworkInterface",
                "ec2:DescribeInstances",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DetachNetworkInterface",
                "ec2:ModifyNetworkInterfaceAttribute",
                "ec2:ResetNetworkInterfaceAttribute"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kinesis:*"
            ],
            "Resource": "arn:aws:kinesis:*:*:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "sns:*"
            ],
            "Resource": "arn:aws:sns:*:*:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "sqs:*"
            ],
            "Resource": "arn:aws:sqs:*:*:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "dynamodb:*"
            ],
            "Resource": "arn:aws:dynamodb:*:*:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "route53:*"
            ],
            "Resource": "*"
        }
    ]
}

静态存储桶ACL策略 #

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowPublicRead",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::xxxx-prd-statics/*"
        }
    ]
}

requirements.txt #

boto3==1.18.5
botocore==1.21.5
certifi==2021.5.30
click==8.0.1
decorator==5.0.9
Django==3.2.3
django-cors-headers==3.7.0
django-extensions==3.1.3
django-filter==2.4.0
django-import-export==2.5.0
djangorestframework==3.12.4
pip-tools==6.2.0
graphene==2.1.9
graphene-file-upload==1.3.0
psycopg2==2.8.6
psycopg2-binary==2.8.6
python-dateutil==2.8.2
PyYAML==5.4.1
requests==2.26.0
Pillow==9.0.1
zappa==0.55.0
PyJWT==v1.7.1
text-unidecode==1.3
django-graphql-jwt==0.3.0
django-graphql-auth==0.3.14
django-admin-interface==0.18.7
django-storages==1.12.3
django-mptt==0.13.4
awscli==1.20.5

解决方案 #

• 区分Zappa部署桶与文件存储桶：确保Django配置中AWS_STORAGE_BUCKET_NAME指向用于存储文件的xxxx-prd-statics，而非Zappa部署用的zappa-xxx-prd，两者用途不同，混淆会导致请求错误。

• 修复存储桶策略资源范围：S3存储桶策略需要同时覆盖桶本身和桶内对象，修改静态存储桶策略的Resource字段为：

  "Resource": [
      "arn:aws:s3:::xxxx-prd-statics",
      "arn:aws:s3:::xxxx-prd-statics/*"
  ]
  

  仅授权对象路径会导致部分操作（如HeadObject）因缺少桶级权限触发错误。

• 移除硬编码密钥：Zappa的Lambda执行角色已配置全量S3权限，无需在zappa_settings.json的环境变量中硬编码AWS密钥。删除以下环境变量，让Lambda使用默认的IAM角色权限，避免密钥冲突或签名错误：

  • AWS_S3_ACCESS_KEY_ID
  • AWS_S3_SECRET_ACCESS_KEY
  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY

• 验证依赖版本兼容性：当前boto3==1.18.5与django-storages==1.12.3可能存在版本适配问题。建议升级django-storages至兼容最新boto3的版本（如django-storages==1.13.2及以上），或调整boto3版本匹配django-storages的要求。

• 检查文件Key格式：确认上传文件的Key没有特殊字符、开头斜杠或格式错误。可在Django配置中添加AWS_LOCATION指定存储前缀，避免Key路径异常：

  AWS_LOCATION = "uploads"
  

内容的提问来源于stack exchange，提问作者Luis Rosero