部署Docker化Python应用至GKE时遇ImagePullBackOff错误

错误日志 #

rpc error: code = Unknown desc = failed to pull and unpack image "us-central1-docker.pkg.dev/myapp-dev-411610/myapp-repo/myapp-app:latest": failed to resolve reference "us-central1-docker.pkg.dev/myapp-dev-xxxxxx/myapp-repo/myapp-app:latest": failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://us-central1-docker.pkg.dev/v2/token?scope=repository%3Amyapp-dev-xxxxxx%2Fmyapp-repo%2Fmyapp-app%3Apull&service=us-central1-docker.pkg.dev: 403 Forbidden

场景说明 #

• 将Docker化Python应用通过Helm部署到GKE集群，镜像托管在GCP Artifact Registry
• 本地Docker可正常构建、推送、拉取该镜像；Cloud Run也能正常拉取该镜像
• GKE环境下，无论使用Helm、kubectl还是GKE UI部署均报上述403错误，即使使用拥有Owner/Organization Administrator权限的主账号操作也无法解决
• 此前使用GCR时遇相同问题，切换至GAR后问题依旧，仅GKE存在该异常

配置文件 #

Terraform配置 #

resource "google_container_cluster" "primary" {
    name = "primary"
    location = "us-central1-b" # region or availablility zone. Should be region for prod
    remove_default_node_pool = true
    initial_node_count = 1
    network = google_compute_network.main.self_link
    subnetwork = google_compute_subnetwork.private.self_link
    logging_service = "logging.googleapis.com/kubernetes"
    monitoring_service = "monitoring.googleapis.com/kubernetes"
    networking_mode = "VPC_NATIVE"

    addons_config {
        http_load_balancing {
            disabled = true
        }
        horizontal_pod_autoscaling {
            disabled = false
        }
    }

    release_channel {
        channel = "REGULAR"
    }

    workload_identity_config {
        workload_pool = "myapp-dev-xxxxxx.svc.id.goog"
    }

    ip_allocation_policy {
        cluster_secondary_range_name = "k8s-pod-range"
        services_secondary_range_name = "k8s-service-range"
    }

    private_cluster_config {
        enable_private_nodes = true
        enable_private_endpoint = false
        master_ipv4_cidr_block = "172.16.0.0/28"
    }
}

resource "google_service_account" "kubernetes" {
    account_id = "kubernetes"
}

resource "google_container_node_pool" "general" {
    name = "general"
    cluster = google_container_cluster.primary.id
    node_count = 1

    management {
        auto_repair = true
        auto_upgrade = true

    }

    node_config {
        preemptible = false
        machine_type = "e2-small"

        labels = {
            role = "general"
        }

        service_account = google_service_account.kubernetes.email
        oauth_scopes = [
            "https://www.googleapis.com/auth/cloud-platform"
        ]
    }
}

resource "google_container_node_pool" "spot" {
    name = "spot"
    cluster = google_container_cluster.primary.id
    node_count = 1

    management {
        auto_repair = true
        auto_upgrade = true

    }

    autoscaling {
        min_node_count = 0
        max_node_count = 10
    }

    node_config {
        preemptible = true # Do not use for prod pool - can be taken offline by GCP at any time
        machine_type = "e2-small"
        
        labels = {
            team = "devops"
        }

        taint {
            key = "instance_type"
            value = "spot"
            effect = "NO_SCHEDULE"
        }

        service_account = google_service_account.kubernetes.email
        oauth_scopes = [
            "https://www.googleapis.com/auth/cloud-platform"
        ]
    }
}

resource "google_service_account" "artifact_registry_service_account" {
  account_id   ="artifact-registry-svc-acct"
  display_name = "Artifact Registry Service Account"
}

resource "google_project_iam_member" "artifact_registry_service_account_reader" {
  project = "myapp-dev-xxxxxx"
  role    = "roles/artifactregistry.reader"
  member  = "serviceAccount:${google_service_account.artifact_registry_service_account.email}"
}

resource "google_service_account_key" "artifact_registry_service_account_key" {
  service_account_id = google_service_account.artifact_registry_service_account.name
  public_key_type    = "TYPE_X509_PEM_FILE"
}

resource "local_file" "artifact_registry_service_account_key_file" {
  content  = base64decode(google_service_account_key.artifact_registry_service_account_key.private_key)
  filename = "${path.module}/artifact-registry-service-account-key.json"
}

resource "kubernetes_secret" "artifact_registry_json_key" {
  metadata {
    name = "artifact-registry-json-key"
  }

  data = {
    ".dockerconfigjson" = jsonencode({
      "auths" = {
        "us-central1-docker.pkg.dev" = {
          "username" = "_json_key"
          "password" = google_service_account_key.artifact_registry_service_account_key.private_key
          "email" = google_service_account.artifact_registry_service_account.email
          "auth" = base64encode("_json_key:${google_service_account_key.artifact_registry_service_account_key.private_key}")
        }
      }
    })
  }
}

resource "google_artifact_registry_repository" "docker_repository" {
  provider = google
  location = "us-central1"
  repository_id = "myapp-repo"
  description = "Docker Repository"
  format = "DOCKER"

  labels = {
    env = "dev"
  }
}

Helm values.yaml配置 #

image:
  repository: us-central1-docker.pkg.dev/myapp-dev-xxxxxx/myapp-repo/myapp-app
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: "latest"

imagePullSecrets:
  - name: artifact-registry-json-key

排查与解决方案 #

1. 修复镜像拉取Secret的格式错误 #

Terraform中创建的kubernetes_secret存在两个关键问题：

• 未指定type = "kubernetes.io/dockerconfigjson"，Kubernetes无法识别为镜像拉取凭证
• 使用了未解码的private_key（GCP返回的私钥是Base64编码的JSON，需解码后才能用作Docker密码）

修正后的Terraform Secret配置：

resource "kubernetes_secret" "artifact_registry_json_key" {
  metadata {
    name = "artifact-registry-json-key"
    namespace = "default" # 需与Helm部署的namespace一致
  }

  type = "kubernetes.io/dockerconfigjson"

  data = {
    ".dockerconfigjson" = jsonencode({
      "auths" = {
        "us-central1-docker.pkg.dev" = {
          "username" = "_json_key"
          "password" = base64decode(google_service_account_key.artifact_registry_service_account_key.private_key)
          "email" = google_service_account.artifact_registry_service_account.email
          "auth" = base64encode("_json_key:${base64decode(google_service_account_key.artifact_registry_service_account_key.private_key)}")
        }
      }
    })
  }
}

2. 给节点服务账号添加Artifact Registry读取权限 #

GKE节点使用的google_service_account.kubernetes服务账号虽配置了cloud-platform scope，但可直接赋予明确的读取权限避免依赖Secret：

resource "google_project_iam_member" "node_sa_artifact_reader" {
  project = "myapp-dev-xxxxxx"
  role    = "roles/artifactregistry.reader"
  member  = "serviceAccount:${google_service_account.kubernetes.email}"
}

3. 验证私有集群的网络访问 #

因集群启用了enable_private_nodes = true，私有节点默认无公网访问能力，需确认：

• 配置Cloud NAT让节点能访问Artifact Registry公网端点
• 或给Artifact Registry配置VPC私有访问，通过内部网络连接GAR

4. 手动验证镜像拉取Secret #

通过CLI创建Secret并测试Pod拉取，排除Terraform配置问题：

# 创建Secret
kubectl create secret docker-registry artifact-registry-json-key \
  --docker-server=us-central1-docker.pkg.dev \
  --docker-username=_json_key \
  --docker-password="$(cat artifact-registry-service-account-key.json)" \
  --docker-email=artifact-registry-svc-acct@myapp-dev-xxxxxx.iam.gserviceaccount.com

# 部署测试Pod
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: test-image-pull
spec:
  containers:
  - name: test-container
    image: us-central1-docker.pkg.dev/myapp-dev-xxxxxx/myapp-repo/myapp-app:latest
  imagePullSecrets:
  - name: artifact-registry-json-key
EOF

若测试Pod能正常启动，说明原Terraform的Secret配置存在格式错误。

内容的提问来源于stack exchange，提问作者Samaruman