GitLab Registry缓存Varnish配置异常：添加Varnish后出现403错误

当前架构：Gitlab Rails → varnish_node:6081 → nginx:80 → OpenStack s3:6780
Rails直接访问Nginx可正常获取S3对象，但通过Varnish转发请求时出现403错误。

相关配置 #

Varnish的default.vcl #

vcl 4.0;

backend default {
    .host = "127.0.0.1";
    .port = "80";
}

sub vcl_recv {
    # 设置客户端IP到自定义头
    set req.http.X-Client-IP = client.ip;
    set req.backend_hint = default;

}

sub vcl_backend_response {
    # 移除或修改可能导致授权问题的头
    unset beresp.http.X-Some-Header;
}

Nginx的nginx.conf #

upstream hpc {
     server hpc.example.com:6780 max_fails=3 fail_timeout=30s;

}
server {
    listen 80 default_server;
        server_name hpc.example.com;

    location / {
                proxy_pass https://hpc;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_http_version 1.1;
                proxy_set_header X-Forwarded-Proto $scheme;   
    }

}

日志信息 #

Nginx日志 #

127.0.0.1 - - [08/May/2024:21:59:12 +0300] "GET /stage-primary-glb-registry?list-type=2&max-keys=1&prefix= HTTP/1.1" 403 193 "-" "docker-distribution/v3.39.2-gitlab (go1.17.6) aws-sdk-go/1.40.7 (go1.17.6; linux; amd64)" "10.20.20.21:6081" "AWS4-HMAC-SHA256 Credential=6f1bfb7443cc45b5a16e5a1cbc1e7976/20240508/mars/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=4a1b957d696d27a9a9d1ca5942e6c241486567a983a2815c78334ac9625002e7" "10.20.20.20"

Varnish日志 #

*   << Request  >> 221
 *   Begin          req 220 rxreq
 *   Timestamp      Start: 1715194922.190385 0.000000 0.000000
 *   Timestamp      Req: 1715194922.190385 0.000000 0.000000
 *   VCL_use        boot
 *   ReqStart       10.20.20.20 35344 a0
 *   ReqMethod      GET
 *   ReqURL         /stage-primary-glb-registry?list-type=2&max-keys=1&prefix=
 *   ReqProtocol    HTTP/1.1
 *   ReqHeader      Host: 10.20.20.21:6081
 *   ReqHeader      User-Agent: docker-distribution/v3.39.2-gitlab (go1.17.6) aws-sdk-go/1.40.7 (go1.17.6; linux; amd64)
 *   ReqHeader      Authorization: AWS4-HMAC-SHA256 Credential=6f1bfb7443cc45b5a16e5a1cbc1e7976/20240508/mars/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=2bbffe23574c9f771fdf87b7ba9823cd5720d301142a9fa015fd09c68d048313
 *   ReqHeader      X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
 *   ReqHeader      X-Amz-Date: 20240508T190202Z
 *   ReqHeader      Accept-Encoding: gzip
 *   ReqHeader      X-Forwarded-For: 10.20.20.20
 *   ReqHeader      Via: 1.1 glb-s-adp-rails-primary-4 (Varnish/7.5)
 *   VCL_call       RECV
 *   ReqHeader      X-Client-IP: 10.20.20.20
 *   VCL_return     pass
 *   VCL_call       HASH
 *   VCL_return     lookup
 *   VCL_call       PASS
 *   VCL_return     fetch
 *   Link           bereq 222 pass
 *   Timestamp      Fetch: 1715194922.257731 0.067345 0.067345
 *   RespProtocol   HTTP/1.1
 *   RespStatus     403
 *   RespReason     Forbidden
 *   RespHeader     Server: nginx/1.18.0 (Ubuntu)
 *   RespHeader     Date: Wed, 08 May 2024 19:02:02 GMT
 *   RespHeader     Content-Type: application/xml
 *   RespHeader     Content-Length: 193
 *   RespHeader     x-amz-request-id: tx0000033f0ecab4cc0e4a6-00663bcc2a-17b6ba11-default
 *   RespHeader     accept-ranges: bytes
 *   RespHeader     X-Varnish: 221
 *   RespHeader     Age: 0
 *   RespHeader     Via: 1.1 glb-s-adp-rails-primary-4 (Varnish/7.5)
 *   VCL_call       DELIVER
 *   VCL_return     deliver
 *   Timestamp      Process: 1715194922.257737 0.067352 0.000006
 *   Filters
 *   RespHeader     Connection: keep-alive
 *   Timestamp      Resp: 1715194922.257767 0.067381 0.000029
 *   ReqAcct        579 0 579 337 193 530
 *   End
 * List item

问题原因与修复方案 #

核心原因 #

从Varnish日志可见，请求经过Varnish时，Host头被修改为10.20.20.21:6081，但AWS4-HMAC-SHA256签名是基于原始域名hpc.example.com生成的。Host头变更导致签名验证失败，S3返回403错误。

直接访问Nginx时，Rails发送的Host头是hpc.example.com，签名验证通过；走Varnish时，Varnish默认保留了客户端发送的自身地址作为Host头，引发签名不匹配。

修复步骤 #

修改Varnish的vcl_recv配置，强制设置正确的Host头：

sub vcl_recv {
    # 强制设置匹配S3签名的Host头
    set req.http.Host = "hpc.example.com";
    set req.http.X-Client-IP = client.ip;
    set req.backend_hint = default;
}

补充说明 #

AWS4签名会将Host头纳入签名参数，任何中间代理修改Host头都会导致签名失效。需确保请求链中Host头与签名时使用的域名一致，直接强制设置正确Host头是最可靠的解决方式。

内容的提问来源于stack exchange，提问作者IeuD