使用Google Cloud SDK上传GCS桶时Kubernetes CronJob报错

核心问题原因 #

1. 只读文件系统报错：google-cloud-cli镜像默认将/home/cloudsdk目录设为只读，gsutil首次运行时需要在该路径下创建.config目录存储配置，触发权限错误。
2. exit code 137：该错误通常是进程因前置报错触发退出，或资源不足被内核终止，解决只读目录问题后大概率会消失。
3. 额外配置问题：原YAML中VAULT_ADDR存在拼写错误（vault.vault-internl.net少了字母a），会导致snapshot容器无法连接Vault。

修复步骤 #

1. 为gsutil提供可写配置目录 #

在upload容器中挂载emptyDir卷到/home/cloudsdk/.config，让gsutil能创建必要的配置文件。

2. 移除冗余sleep命令 #

upload容器的sleep 500属于冗余逻辑，保留until循环等待备份文件即可。

3. 修正Vault地址拼写错误 #

将snapshot容器的VAULT_ADDR值改为正确的https://vault.vault-internal.net:8200。

4. 确保GCS权限配置正确 #

• 若使用GKE Workload Identity，需将K8s ServiceAccount raft-backup绑定到拥有GCS桶写入权限的GCP ServiceAccount。
• 若使用密钥挂载，需将GCP服务账号密钥以Secret形式挂载到upload容器，并设置GOOGLE_APPLICATION_CREDENTIALS环境变量指向密钥路径。

修改后的完整CronJob YAML #

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  labels:
    app.kubernetes.io/component: raft-backup
    numenapp: raft-backup
  name: raft-backup
  namespace: raft-backup
spec:
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        metadata:
          annotations:
            vault.security.banzaicloud.io/vault-addr: https://vault.vault-internal.net:8200
          labels:
            app.kubernetes.io/component: raft-backup
        spec:
          containers:
          - args:
              - |
                SA_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token);
                export VAULT_TOKEN=$(vault write -field=token auth/kubernetes/login jwt=$SA_TOKEN role=raft-backup);
                vault operator raft snapshot save /share/vault-raft.snap;
                echo "snapshot is success"
            command: ["/bin/sh", "-c"]
            env:
            - name: VAULT_ADDR
              value: https://vault.vault-internal.net:8200  # 修正拼写错误
            image: vault:1.10.9
            imagePullPolicy: Always
            name: snapshot
            volumeMounts:
            - mountPath: /share
              name: share
          - args:
            - -ec
            - |
              until [ -f /share/vault-raft.snap ]; do sleep 5; done;
              gsutil cp /share/vault-raft.snap gs://raft-backup/vault_raft_$(date +"%Y%m%d_%H%M%S").snap;
            command:
            - /bin/sh
            image: gcr.io/google.com/cloudsdktool/google-cloud-cli:latest
            imagePullPolicy: IfNotPresent
            name: upload
            securityContext:
              allowPrivilegeEscalation: false
            volumeMounts:
            - mountPath: /share
              name: share
            - mountPath: /home/cloudsdk/.config  # 添加可写配置目录挂载
              name: gsutil-config
          restartPolicy: OnFailure
          securityContext:
            fsGroup: 1000
            runAsGroup: 1000
            runAsUser: 1000
          serviceAccountName: raft-backup
          volumes:
          - emptyDir: {}
            name: share
          - emptyDir: {}  # 新增emptyDir卷用于gsutil配置存储
            name: gsutil-config
  schedule: '*/3 * * * *'
  startingDeadlineSeconds: 60
  successfulJobsHistoryLimit: 3
  suspend: false

验证方法 #

1. 应用修改后的配置：kubectl apply -f raft-backup-cronjob.yaml -n raft-backup
2. 等待CronJob触发任务后，查看upload容器日志：kubectl logs <upload-pod-name> -n raft-backup
3. 检查GCS桶内是否生成对应备份文件，确认上传流程正常。

内容的提问来源于stack exchange，提问作者kapil dev