新建GKE集群无法拉取Artifact Registry私有镜像问题排查

问题场景 #

通过Terraform使用自定义服务账号创建GKE集群后，执行Helm部署应用时出现镜像拉取失败，Pod状态为ImagePullBackOff。

集群创建的Terraform代码 #

locals {
  service         = "example"
  resource_prefix = format("%s-%s", local.service, var.env)
  location        = format("%s-b", var.gcp_region)
}

resource "google_service_account" "main" {
  account_id   = format("%s-sa", local.resource_prefix)
  display_name = format("%s-sa", local.resource_prefix)
  project      = var.gcp_project
}

resource "google_container_cluster" "main" {
  name                     = local.resource_prefix
  description              = format("Cluster primarily servicing the service %s", local.service)
  location                 = local.location
  remove_default_node_pool = true
  initial_node_count       = 1
}

resource "google_container_node_pool" "main" {
  name       = format("%s-node-pool", local.resource_prefix)
  location   = local.location
  cluster    = google_container_cluster.main.name
  node_count = var.gke_cluster_node_count

  node_config {
    preemptible  = true
    machine_type = var.gke_node_machine_type
    # Google recommends custom service accounts that have cloud-platform scope and permissions granted via IAM Roles.
    service_account = google_service_account.main.email
    oauth_scopes = [
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring",
      "https://www.googleapis.com/auth/cloud-platform",
      "https://www.googleapis.com/auth/devstorage.read_only",
      "https://www.googleapis.com/auth/servicecontrol",
      "https://www.googleapis.com/auth/service.management.readonly",
      "https://www.googleapis.com/auth/trace.append"
    ]
  }

  autoscaling {
    min_node_count = var.gke_cluster_autoscaling_min_node_count
    max_node_count = var.gke_cluster_autoscaling_max_node_count
  }
}

错误信息 #

Pod状态：

default       php-5996c7fbfd-d6xf5                                             0/1     ImagePullBackOff             0          37m

详细报错：

Normal   Pulling    36m (x4 over 37m)      kubelet            Pulling image "europe-docker.pkg.dev/example-999999/eu.gcr.io/example-php-fpm:latest"
  Warning  Failed     36m (x4 over 37m)      kubelet            Failed to pull image "europe-docker.pkg.dev/example-999999/eu.gcr.io/example-php-fpm:latest": rpc error: code = Unknown desc = failed to pull and unpack image "europe-docker.pkg.dev/example-999999/eu.gcr.io/example-php-fpm:latest": failed to resolve reference "europe-docker.pkg.dev/example-999999/eu.gcr.io/example-php-fpm:latest": failed to authorize: failed to fetch oauth token: unexpected status: 403 Forbidden
  Warning  Failed     36m (x4 over 37m)      kubelet            Error: ErrImagePull
  Warning  Failed     35m (x6 over 37m)      kubelet            Error: ImagePullBackOff

经排查，问题与服务账号权限相关，即使通过Cloud SSH生成OAuth token，使用crictl仍无法拉取镜像。

解决方法 #

为自定义服务账号添加Artifact Registry读取权限，在Terraform中新增以下资源配置：

resource "google_project_iam_member" "artifact_role" {
  role = "roles/artifactregistry.reader"
  member  = "serviceAccount:${google_service_account.main.email}"
  project = var.gcp_project
}

添加后重新部署Terraform，集群节点即可正常拉取Artifact Registry中的私有镜像。

内容的提问来源于stack exchange，提问作者David