Nginx反向代理下s3gw部署后AWS CLI创建Bucket失败排查

背景 #

本地部署s3gw后操作一切正常，但部署到Droplet并通过Nginx反向代理后，使用AWS CLI创建Bucket时持续返回403错误。本地环境执行相同操作无异常，生产环境的S3_ACCESS_KEY和S3_SECRET_KEY环境变量已正确配置且可正常读取。

本地Docker Compose配置 #

services:
  s3:
    image: quay.io/s3gw/s3gw
    volumes:
      - dev-s3:/data
    environment:
      RGW_DEFAULT_USER_ACCESS_KEY: dev
      RGW_DEFAULT_USER_SECRET_KEY: dev
    ports:
      - 7480:7480
    restart: on-failure

volumes:
  mssql-datavolume:
    driver: local
  dev-s3:

生产环境Docker Compose配置 #

services:
  s3:
    image: quay.io/s3gw/s3gw
    volumes:
      - dev-s3:/data
    environment:
      RGW_DEFAULT_USER_ACCESS_KEY: ${S3_ACCESS_KEY}
      RGW_DEFAULT_USER_SECRET_KEY: ${S3_SECRET_KEY}
    ports:
      - "7480:7480"
    restart: on-failure
    networks:
     - test-network
  
  nginx:
    image: nginx:stable-alpine
    ports:
      - "80:80"
      - "443:443"
    environment:
      SERVER_NAME: ${SERVER_NAME}
      S3_URL: ${S3_URL}
    volumes:
      - ./deploy/nginx/templates:/etc/nginx/templates:ro
      - ./etc/letsencrypt:/etc/letsencrypt:ro
      - ./certbot/data:/var/www/certbot
    depends_on:
      - s3
    networks:
      - test-network

volumes:
  mssql-datavolume:
    driver: local
  dev-s3:

networks:
  test-network:
    driver: bridge

Nginx模板配置 #

server {
        listen [::]:80;
        listen 80;
        server_name ${SERVER_NAME} www.${SERVER_NAME};
        return 301 https://$host$request_uri;
    }

server {
        listen [::]:443 ssl;
        listen 443 ssl;
        server_name ${SERVER_NAME} www.${SERVER_NAME};
         
        ssl_certificate /etc/letsencrypt/live/${SERVER_NAME}/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/${SERVER_NAME}/privkey.pem;
        include /etc/letsencrypt/options-ssl-nginx.conf;
        ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
    
        location ~ /.well-known/acme-challenge {
            allow all;
            root /var/www/certbot;
        }        

        location / {
            proxy_pass ${UI_URL};
            proxy_set_header Host $host;
            proxy_pass_request_headers on;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection 'upgrade';
        }
        
        location /s3/ {
            rewrite ^/s3/(.*)$ /$1 break;
            proxy_pass ${S3_URL};
            proxy_http_version     1.1;
            proxy_redirect off;
            proxy_set_header       Connection "";
            proxy_set_header       Authorization '';
            proxy_set_header       Host s3;
            proxy_set_header       X-Real-IP $remote_addr;
            proxy_set_header       X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_hide_header      x-amz-id-2;
            proxy_hide_header      x-amz-request-id;
            proxy_hide_header      x-amz-meta-server-side-encryption;
            proxy_hide_header      x-amz-server-side-encryption;
            proxy_hide_header      Set-Cookie;
            proxy_ignore_headers   Set-Cookie;
            proxy_intercept_errors on;
            add_header             Cache-Control max-age=31536000;
        }
    }

Nginx容器日志 #

2024-10-11T19:34:31.849+0000 7a6126a00700  1 beast: 0x7a6154924770: 172.19.0.8 - - [11/Oct/2024:19:34:31.845 +0000] "HEAD /test-articles HTTP/1.1" 403 0 - "aws-cli/1.35.1 md/Botocore#1.35.35 ua/2.0 os/linux#6.8.0-45-generic md/arch#x86_64 lang/python#3.12.3 md/pyimpl#CPython cfg/retry-mode#legacy botocore/1.35.35" - latency=0.003999994s
2024-10-11T19:34:36.291+0000 7a6120600700  1 beast: 0x7a6154924770: 172.19.0.8 - - [11/Oct/2024:19:34:36.290 +0000] "PUT /test-articles HTTP/1.1" 403 120 - "aws-cli/1.35.1 md/Botocore#1.35.35 ua/2.0 os/linux#6.8.0-45-generic md/arch#x86_64 lang/python#3.12.3 md/pyimpl#CPython cfg/retry-mode#legacy botocore/1.35.35" - latency=0.000999999s
2024-10-11T19:34:40.407+0000 7a6112a00700  1 beast: 0x7a6154924770: 172.19.0.8 - - [11/Oct/2024:19:34:40.393 +0000] "PUT /test-articles?policy HTTP/1.1" 403 120 - "aws-cli/1.35.1 md/Botocore#1.35.35 ua/2.0 os/linux#6.8.0-45-generic md/arch#x86_64 lang/python#3.12.3 md/pyimpl#CPython cfg/retry-mode#legacy botocore/1.35.35" - latency=0.013999981s

问题分析与修复方案 #

核心问题 #

1. Nginx配置中proxy_set_header Authorization '';直接清空了请求的Authorization头部，而AWS CLI的身份签名正是通过该头部传递的，导致s3gw无法验证请求身份，返回403错误。
2. proxy_set_header Host s3;设置的Host头与s3gw签名验证逻辑不匹配，S3协议的签名验证依赖请求的实际域名Host头。

修复步骤 #

1. 删除Authorization头部清空配置
   移除Nginx /s3/ location块中的proxy_set_header Authorization '';，确保签名信息能完整传递给s3gw。

2. 调整Host头配置
   将proxy_set_header Host s3;修改为proxy_set_header Host $host;，保持与外部请求一致的域名Host头，符合S3签名验证要求。

3. 补充X-Forwarded-Proto头
   添加以下配置到/s3/ location块，告知s3gw原始请求使用HTTPS协议，避免签名验证时的协议不匹配：

   proxy_set_header X-Forwarded-Proto $scheme;
   

4. 重启Nginx生效配置
   执行命令重启Nginx容器：

   docker-compose restart nginx
   

5. 验证AWS CLI命令
   使用正确的endpoint执行创建Bucket命令：

   aws s3 mb s3://test-articles --endpoint-url https://your-domain.com/s3/
   

内容的提问来源于stack exchange，提问作者Lev Kostychenko