云原生Gitlab Registry S3存储配置异常：blob upload unknown

问题现象 #

在AWS EKS环境（搭配NGINX Ingress）通过GitLab云原生Helm Chart部署GitLab，主对象存储（代码包、LFS等）运行正常，但Registry使用S3作为对象存储时，docker登录认证成功，但推送镜像时始终返回blob upload unknown的404错误。

错误日志 #

{"content_type":"application/json","correlation_id":"41ffa7b8b1e69bcd20d10a962cbce957","duration_ms":1,"host":"self-hosted-redacted.com","level":"info","method":"PATCH","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"10.4.73.112:41348","remote_ip":"10.4.73.112","status":404,"system":"http","time":"2024-09-09T03:43:49.489Z","ttfb_ms":0,"uri":"/v2/repo/repo/elx-tutorial/blobs/uploads/c70be522-6bb7-408b-9478-1eade0e45b18?_state=7ls38sWs9i_lrMjKq5Gs8ydBIJxpYHRGVskCcPq0SRN7Ik5hbWUiOiIwNzJfY3MvMDcyLW1vbm9yZXBvL2VseC10dXRvcmlhbCIsIlVVSUQiOiJjNzBiZTUyMi02YmI3LTQwOGItOTQ3OC0xZWFkZTBlNDViMTgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjQtMDktMDlUMDM6NDM6NDkuMTg5NzMwNDgyWiJ9","user_agent":"docker/26.1.4 go/go1.21.11 git-commit/de5c9cf kernel/6.6.31-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/26.1.4 (darwin))","written_bytes":100}

尝试过的配置形式 #

1. Registry官方文档子配置 #

registry:
  enabled:
  maintenance:
    readonly:
      enabled: false
    uploadpurging:
      enabled: true
      age: 168h
      interval: 24h
      dryrun: false
  image:
    tag: 'v4.8.0-gitlab'
    pullPolicy: IfNotPresent
  annotations:
  service:
    type: ClusterIP
    name: registry
  httpSecret:
    secret:
    key:
  authEndpoint:
  tokenIssuer:
  certificate:
    secret: gitlab-registry
    key: registry-auth.crt
  deployment:
    terminationGracePeriodSeconds: 30
  draintimeout: '0'
  hpa:
    minReplicas: 2
    maxReplicas: 10
    cpu:
      targetAverageUtilization: 75
    behavior:
      scaleDown:
        stabilizationWindowSeconds: 300
  storage:
    secret:
    key: storage
    extraKey:
  validation:
    disabled: true
    manifests:
      referencelimit: 0
      payloadsizelimit: 0
      urls:
        allow: []
        deny: []
  notifications: {}
  tolerations: []
  affinity: {}
  ingress:
    enabled: false
    tls:
      enabled: true
      secretName: redis
    annotations:
    configureCertmanager:
    proxyReadTimeout:
    proxyBodySize:
    proxyBuffering:
  networkpolicy:
    enabled: false
    egress:
      enabled: false
      rules: []
    ingress:
      enabled: false
      rules: []
  serviceAccount:
    create: false
    automountServiceAccountToken: false
  tls:
    enabled: false
    secretName:
    verify: true
    caSecretName:

2. GitLab Chart默认values中的Registry配置 #

## https://docs.gitlab.com/charts/charts/globals#configure-registry-settings
registry:
  bucket: registry

  certificate: {}
    # secret:
  httpSecret: {}
    # secret:
    # key:
  notificationSecret: {}
    # secret:
    # key:
  tls:
    enabled: false
    # secretName:
  redis:
    cache:
      password: {}
    rateLimiting:
      password: {}
  # https://docs.docker.com/registry/notifications/#configuration
  notifications: {}
    # endpoints:
    #   - name: FooListener
    #     url: https://foolistener.com/event
    #     timeout: 500ms
    #     threshold: 10 # DEPRECATED: use maxretries instead https://gitlab.com/gitlab-org/container-registry/-/issues/1243.
    #     maxretries: 5
    #     backoff: 1s
    #     headers:
    #       FooBar: ['1', '2']
    #       Authorization:
    #         secret: gitlab-registry-authorization-header
    #       SpecificPassword:
    #         secret: gitlab-registry-specific-password
    #         key: password
    # events: {}

  # Settings utilized by other services referencing registry:
  enabled: true
  host:
  # port: 443
  api:
    protocol: http
    serviceName: registry
    port: 5000
  tokenIssuer: gitlab-issuer

3. S3驱动存储配置示例 #

s3:
  bucket: gitlab-registry-storage
  accesskey: AWS_ACCESS_KEY
  secretkey: AWS_SECRET_KEY
  region: us-east-1
  # regionendpoint: "https://minio.example.com:9000"
  v4auth: true

对应Registry存储引用配置：

storage:
  secret: gitlab-object-storage
  key: connection

当前对象存储配置 #

S3连接配置（存储在gitlab-object-storage Secret中） #

provider: AWS
region: us-gov-west-1
aws_access_key_id: <your-access-key>
aws_secret_access_key: <your-secret-key>

GitLab全局object_store配置 #

object_store:
  enabled: true
  proxy_download: true
  storage_options: {}
    # server_side_encryption:
  # server_side_encryption_kms_key_id
  connection:
    secret: gitlab-object-storage

解决方案建议 #

1. 修正GovCloud区域的S3端点配置 #

AWS GovCloud（us-gov-west-1）的S3端点与公有云不同，需要显式指定regionendpoint：
在S3连接配置中添加：

regionendpoint: https://s3.us-gov-west-1.amazonaws.com

确保Registry能正确访问GovCloud的S3服务。

2. 确保Registry的storage配置与全局object_store关联正确 #

避免配置冲突，使用统一的存储Secret关联：

global:
  registry:
    bucket: gitlab-registry-storage
    storage:
      secret: gitlab-object-storage
      key: connection

registry:
  storage:
    secret: gitlab-object-storage
    key: connection

3. 验证S3权限 #

确认用于Registry的IAM用户/角色拥有目标bucket的以下权限：

• s3:ListBucket
• s3:GetObject
• s3:PutObject
• s3:DeleteObject
  同时检查桶策略是否存在访问限制。

4. 检查Redis缓存配置 #

Registry依赖Redis处理上传会话，确保：

• global.registry.redis.cache.password配置正确，能正常访问GitLab部署的Redis实例
• Registry pod日志中无Redis连接失败报错

5. 验证Registry的httpSecret一致性 #

确保全局和Registry子配置的httpSecret使用同一个Secret，避免会话验证失败：

global:
  registry:
    httpSecret:
      secret: gitlab-registry-secrets
      key: httpSecret

registry:
  httpSecret:
    secret: gitlab-registry-secrets
    key: httpSecret

6. 调整NGINX Ingress配置 #

• 设置足够大的proxy_read_timeout（建议至少120s），避免大镜像上传超时：

registry:
  ingress:
    proxyReadTimeout: 120s

• 确认Ingress未拦截PATCH请求（Registry上传依赖该方法）。

内容的提问来源于stack exchange，提问作者Steve Nadraus