GKE Autopilot集群GPU任务部署遇CPU/内存不足及扩容失败问题

问题背景 #

在GKE Autopilot集群部署GPU密集型Job时，出现CPU/内存不足的调度失败错误，集群尝试自动扩容但失败，kubectl get nodes显示现有3个节点剩余资源仅约0.5GB CPU和内存。

调度事件日志 #

Warning  FailedScheduling   27m (x5 over 31m)      gke.io/optimize-utilization-scheduler  0/2 nodes are available: 2 Insufficient cpu, 2 Insufficient memory.
Warning  FailedScheduling   26m                    gke.io/optimize-utilization-scheduler  0/3 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate, 2 Insufficient cpu, 2 Insufficient memory.
Normal   TriggeredScaleUp   26m                    cluster-autoscaler                     pod triggered scale-up: [{https://www.googleapis.com/compute/v1/projects/picdmo-342711/zones/us-central1-c/instanceGroups/gk3-picdmo-nap-1wcisjk4-2ba03e97-grp 0->1 (max: 1000)}]
Normal   NotTriggerScaleUp  25m (x6 over 30m)      cluster-autoscaler                     pod didn't trigger scale-up (it wouldn't fit if a new node is added): 1 Insufficient cpu, 1 Insufficient memory, 2 in backoff after failed scale-up
Normal   NotTriggerScaleUp  20m (x14 over 21m)     cluster-autoscaler                     pod didn't trigger scale-up (it wouldn't fit if a new node is added): 2 Insufficient cpu, 1 Insufficient memory
Normal   TriggeredScaleUp   15m                    cluster-autoscaler                     pod triggered scale-up: [{https://www.googleapis.com/compute/v1/projects/picdmo-342711/zones/us-central1-c/instanceGroups/gk3-picdmo-nap-xt7d8ijc-37c84d94-grp 0->1 (max: 1000)}]
Warning  FailedScaleUp      15m (x5 over 31m)      cluster-autoscaler                     Node scale up in zones us-central1-c associated with this pod failed: GCE quota exceeded. Pod is at risk of not being scheduled.
Warning  FailedScheduling   15m (x6 over 20m)      gke.io/optimize-utilization-scheduler  0/4 nodes are available: 3 Insufficient memory, 4 Insufficient cpu.
Normal   NotTriggerScaleUp  14m (x2 over 15m)      cluster-autoscaler                     (combined from similar events): pod didn't trigger scale-up (it wouldn't fit if a new node is added): 2 in backoff after failed scale-up, 2 Insufficient cpu, 1 Insufficient memory
Warning  FailedScheduling   13m (x2 over 14m)      gke.io/optimize-utilization-scheduler  0/4 nodes are available: 1 node(s) had taint {ToBeDeletedByClusterAutoscaler: 1660665555}, that the pod didn't tolerate, 3 Insufficient cpu, 3 Insufficient memory.
Normal   NotTriggerScaleUp  4m50s (x135 over 29m)  cluster-autoscaler                     pod didn't trigger scale-up (it wouldn't fit if a new node is added): 1 Insufficient cpu, 1 Insufficient memory
Warning  FailedScheduling   92s (x17 over 25m)     gke.io/optimize-utilization-scheduler  0/3 nodes are available: 3 Insufficient cpu, 3 Insufficient memory.

部署的Job配置 #

apiVersion: batch/v1
kind: Job
metadata:
  generateName: asd-job-
spec:
  template:
    spec:
      containers:
        - name: asd
          image: gcr.io/-342711/-job:latest
          imagePullPolicy: Always
          command: ["/bin/sh"]
          args: ["-c", "echo"]
          resources:
            requests:
              memory: "16000Mi"
              cpu: "8000m"
            limits:
              memory: "32000Mi"
              cpu: "16000m"
              nvidia.com/gpu: 2
      restartPolicy: Never
  backoffLimit: 4

问题原因分析 #

1. GCE资源配额耗尽：日志中明确出现FailedScaleUp: GCE quota exceeded，说明集群尝试扩容GPU节点时，项目在us-central1-c区域的GPU、CPU或内存配额不足，无法创建新节点。
2. Pod资源请求与节点规格不匹配：Pod请求8核CPU、16G内存+2块GPU，但Autopilot自动选择的GPU节点类型可能无法满足该资源需求（或节点预留系统资源后剩余空间不足），导致即使扩容新节点也无法容纳Pod，出现pod didn't trigger scale-up (it wouldn't fit if a new node is added)。
3. 节点调度污点未被容忍：部分节点带有node.kubernetes.io/not-ready或ToBeDeletedByClusterAutoscaler污点，而Pod未配置对应的容忍度，进一步减少了可用节点数量。

解决方案 #

1. 检查并提升GCE资源配额 #

• 登录Google Cloud控制台，进入IAM与管理 > 配额页面，筛选区域us-central1-c，重点检查以下配额：
  • 特定GPU类型的配额（如NVIDIA T4/V100等）
  • 对应机器系列的CPU配额（如n1系列CPU配额）
  • 内存配额
• 若配额不足，点击配额旁的“编辑配额”提交提升申请，等待Google Cloud审核通过。

2. 调整Pod资源请求，适配Autopilot节点规格 #

• Autopilot的GPU节点常用规格：n1-standard-16（16核60G内存）支持挂载2块GPU，但节点会预留约20%资源给系统组件，实际可用CPU约12.8核、内存约48G。
• 调整Pod的资源请求至节点可用范围内，比如：

  resources:
    requests:
      memory: "14000Mi"
      cpu: "7000m"
    limits:
      memory: "30000Mi"
      cpu: "14000m"
      nvidia.com/gpu: 2
  

3. 指定GPU节点的机器类型（可选） #

• 在Pod spec中添加nodeSelector，指定支持2块GPU的机器类型，确保Autopilot扩容时创建匹配的节点：

  spec:
    template:
      spec:
        nodeSelector:
          cloud.google.com/gke-machine-type: n1-standard-16
        containers:
        # ... 其余配置不变
  

4. 添加节点污点容忍度（可选） #

• 若需要使用带有ToBeDeletedByClusterAutoscaler污点的节点，可在Pod spec中添加容忍度：

  spec:
    template:
      spec:
        tolerations:
        - key: "ToBeDeletedByClusterAutoscaler"
          operator: "Exists"
          effect: "NoSchedule"
        containers:
        # ... 其余配置不变
  

内容的提问来源于stack exchange，提问作者D. Richard