通过Kubernetes Ingress访问非公开GCP Storage Bucket的实现方案

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: assets-ingress
namespace: myservice
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /bucket_name/$1
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/upstream-vhost: storage.googleapis.com
nginx.ingress.kubernetes.io/auth-url: http://myserver.com/auth/users/accessToken
nginx.ingress.kubernetes.io/auth-response-headers: Authorization
spec:
ingressClassName: nginx
rules:
- host: '.myserver.com'
http:
paths:
- path: "/assets/(.)"
pathType: Prefix
backend:
service:
name: google-storage-buckets-service
port:
number: 443

当前配置需要将存储桶设为公开才能生效，我希望关闭存储桶的公开访问权限，同时保留Ingress的`auth-url`权限验证，保持Ingress正常工作。

---

## 解决方案
要实现非公开GCS存储桶的Ingress访问，核心是让Nginx Ingress转发请求到GCS时携带有效的GCP身份凭证，具体操作步骤如下：

### 1. 创建并配置GCP服务账号
- 在GCP控制台创建服务账号（命名示例：`gcs-ingress-proxy`）
- 给该账号授予**Storage Object Viewer**（仅读）或**Storage Object Admin**（读写）权限，权限范围限定到目标存储桶
- 下载该服务账号的JSON密钥文件

### 2. 将密钥文件存入Kubernetes Secret
执行以下命令，在`myservice`命名空间下创建Secret存储密钥：
```bash
kubectl create secret generic gcs-sa-secret --from-file=service-account.json=/path/to/your/downloaded-key.json -n myservice

3. 更新Ingress配置，添加GCS认证逻辑 #

修改Ingress资源，添加自定义Nginx配置，让它自动生成GCP访问令牌并添加到转发请求头：

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: assets-ingress
  namespace: myservice
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /bucket_name/$1
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/upstream-vhost: storage.googleapis.com
    nginx.ingress.kubernetes.io/auth-url: http://myserver.com/auth/users/accessToken
    nginx.ingress.kubernetes.io/auth-response-headers: Authorization
    # GCS认证相关配置
    nginx.ingress.kubernetes.io/configuration-snippet: |
      lua_need_request_body on;
      set $gcs_token "";
      access_by_lua_block {
        local cjson = require "cjson"
        local jwt = require "resty.jwt"
        -- 读取挂载的服务账号密钥
        local sa_file = io.open("/etc/nginx/secrets/gcs-sa-secret/service-account.json")
        local sa = cjson.decode(sa_file:read("*a"))
        sa_file:close()
        -- 构造JWT payload
        local payload = {
          iss = sa.client_email,
          scope = "https://www.googleapis.com/auth/devstorage.read_only",
          aud = "https://www.googleapis.com/oauth2/v4/token",
          exp = os.time() + 3600, -- 令牌有效期1小时
          iat = os.time()
        }
        -- 生成JWT签名
        local token = jwt:sign(sa.private_key, {
          header = { alg = "RS256", typ = "JWT" },
          payload = payload
        })
        -- 请求GCP获取临时访问令牌
        local res = ngx.location.capture("/_gcs_token", {
          method = ngx.HTTP_POST,
          body = "grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer&assertion=" .. token
        })
        if res.status == 200 then
          local data = cjson.decode(res.body)
          ngx.var.gcs_token = data.access_token
        end
      }
    nginx.ingress.kubernetes.io/server-snippet: |
      # 内部请求GCP OAuth2令牌端点
      location /_gcs_token {
        internal;
        proxy_pass https://www.googleapis.com/oauth2/v4/token;
        proxy_set_header Content-Type application/x-www-form-urlencoded;
      }
    # 将令牌添加到转发请求头
    nginx.ingress.kubernetes.io/proxy-set-header: |
      Authorization "Bearer $gcs_token"
spec:
  ingressClassName: nginx
  rules:
    - host: '*.myserver.com'
      http:
        paths:
          - path: "/assets/(.*)"
            pathType: Prefix          
            backend:
              service:
                name: google-storage-buckets-service
                port:
                  number: 443

4. 配置Nginx Ingress Controller挂载Secret #

如果你的Ingress Controller是通过Helm安装的，需要修改Helm values，添加Secret的挂载配置：

extraVolumes:
  - name: gcs-sa-secret
    secret:
      secretName: gcs-sa-secret
extraVolumeMounts:
  - name: gcs-sa-secret
    mountPath: /etc/nginx/secrets/gcs-sa-secret
    readOnly: true

然后执行Helm升级命令：

helm upgrade ingress-nginx ingress-nginx/ingress-nginx -f your-values.yaml -n ingress-nginx

5. 关闭存储桶公开访问 #

进入GCP控制台的存储桶页面，移除所有针对allUsers或allAuthenticatedUsers的公开权限条目，确保存储桶仅允许授权访问。

内容的提问来源于stack exchange，提问作者Mariusz