GKE集群Job因临时存储不足被驱逐，如何通过Django配置修复？

我有一个GKE集群，通过Django创建Kubernetes Job运行C++代码镜像，构建流程由GitHub触发。之前一切正常，但最近推送了一个仅包含三四行基础操作的小提交后，镜像能正常构建，但创建Job时出现报错：Pod errors: BackoffLimitExceeded, Error with exit code 137，Job无法完成。

通过kubectl describe POD_NAME排查失败Pod，关键输出如下：

Events:
  Type     Reason               Age    From               Message
  ----     ------               ----   ----               -------
  Normal   Scheduled            7m32s  default-scheduler  Successfully assigned default/xvb8zfzrhhmz-jk9vf to gke-cluster-1-default-pool-ee7e99bb-xzhk
  Normal   Pulling              7m7s   kubelet            Pulling image "gcr.io/videoo3-360019/github.com/videoo-io/videoo-render:latest"
  Normal   Pulled               4m1s   kubelet            Successfully pulled image "gcr.io/videoo3-360019/github.com/videoo-io/videoo-render:latest" in 3m6.343917225s
  Normal   Created              4m1s   kubelet            Created container jobcontainer
  Normal   Started              4m     kubelet            Started container jobcontainer
  Warning  Evicted              3m29s  kubelet            The node was low on resource: ephemeral-storage. Container jobcontainer was using 91144Ki, which exceeds its request of 0.
  Normal   Killing              3m29s  kubelet            Stopping container jobcontainer
  Warning  ExceededGracePeriod  3m19s  kubelet            Container runtime did not kill the pod within specified grace period.

核心错误日志：

The node was low on resource: ephemeral-storage. Container jobcontainer was using 91144Ki, which exceeds its request of 0.

我没有用YAML配置Pod，而是通过Django调用Kubernetes API创建Job，相关代码如下：

def kube_create_job_object(name, container_image, namespace="default", container_name="jobcontainer", env_vars={}):
    # Body is the object Body
    body = client.V1Job(api_version="batch/v1", kind="Job")
    # Body needs Metadata
    # Attention: Each JOB must have a different name!
    body.metadata = client.V1ObjectMeta(namespace=namespace, name=name)
    # And a Status
    body.status = client.V1JobStatus()
     # Now we start with the Template...
    template = client.V1PodTemplate()
    template.template = client.V1PodTemplateSpec()
    # Passing Arguments in Env:
    env_list = []
    for env_name, env_value in env_vars.items():
        env_list.append( client.V1EnvVar(name=env_name, value=env_value) )

    print(env_list)
    security = client.V1SecurityContext(privileged=True, allow_privilege_escalation=True, capabilities= client.V1Capabilities(add=["CAP_SYS_ADMIN"]))
    container = client.V1Container(name=container_name, image=container_image, env=env_list, stdin=True, security_context=security)
    template.template.spec = client.V1PodSpec(containers=[container], restart_policy='Never')
    body.spec = client.V1JobSpec(backoff_limit=0, ttl_seconds_after_finished=600, template=template.template)
    return body



def kube_create_job(manifest, output_uuid, output_signed_url, webhook_url, valgrind, sleep, isaudioonly):
    credentials, project = google.auth.default(
        scopes=['https://www.googleapis.com/auth/cloud-platform', ])
    credentials.refresh(google.auth.transport.requests.Request())
    cluster_manager = ClusterManagerClient(credentials=credentials)
    cluster = cluster_manager.get_cluster(name=f"path/to/cluster")

    with NamedTemporaryFile(delete=False) as ca_cert:
        ca_cert.write(base64.b64decode(cluster.master_auth.cluster_ca_certificate))

    config = client.Configuration()
    config.host = f'https://{cluster.endpoint}:443'
    config.verify_ssl = True
    config.api_key = {"authorization": "Bearer " + credentials.token}
    config.username = credentials._service_account_email
    config.ssl_ca_cert = ca_cert.name
    client.Configuration.set_default(config)

    # Setup K8 configs
    api_instance = kubernetes.client.BatchV1Api(kubernetes.client.ApiClient(config))

    container_image = get_first_success_build_from_list_builds(client)
    name = id_generator()
    body = kube_create_job_object(name, container_image,
                                  env_vars={
                                            "PROJECT"           : json.dumps(manifest),
                                            "BUCKET"            : settings.GS_BUCKET_NAME,
                                           })
    try:
        api_response = api_instance.create_namespaced_job("default", body, pretty=True)
        print(api_response)
    except ApiException as e:
        print("Exception when calling BatchV1Api->create_namespaced_job: %s\n" % e)
    return body

• 你的Pod属于BestEffort QoS类（未设置任何资源请求/限制），当节点临时存储（ephemeral-storage）不足时，Kubernetes会优先驱逐这类Pod。
• 容器实际使用了约91MiB临时存储，但你未设置ephemeral-storage的请求值（默认为0），触发了节点的存储资源阈值，导致Pod被kubelet强制终止（exit code 137是容器被强制杀死的典型标志）。

必须为容器设置临时存储的资源请求（request）和限制（limit）：

1. 资源请求（request）：告诉Kubernetes调度器，Pod需要至少多少临时存储，调度器只会把Pod分配到有足够剩余存储的节点。
2. 资源限制（limit）：限制容器最多能使用多少临时存储，防止容器过度占用节点资源影响其他Pod。

额外可检查GKE节点的临时存储使用情况，必要时扩容节点池或清理节点上的临时文件。

修改kube_create_job_object函数，为容器添加资源配置：

def kube_create_job_object(name, container_image, namespace="default", container_name="jobcontainer", env_vars={}):
    # Body is the object Body
    body = client.V1Job(api_version="batch/v1", kind="Job")
    # Body needs Metadata
    # Attention: Each JOB must have a different name!
    body.metadata = client.V1ObjectMeta(namespace=namespace, name=name)
    # And a Status
    body.status = client.V1JobStatus()
     # Now we start with the Template...
    template = client.V1PodTemplate()
    template.template = client.V1PodTemplateSpec()
    # Passing Arguments in Env:
    env_list = []
    for env_name, env_value in env_vars.items():
        env_list.append( client.V1EnvVar(name=env_name, value=env_value) )

    print(env_list)
    security = client.V1SecurityContext(privileged=True, allow_privilege_escalation=True, capabilities= client.V1Capabilities(add=["CAP_SYS_ADMIN"]))
    
    # 添加资源请求和限制配置
    resources = client.V1ResourceRequirements(
        requests={"ephemeral-storage": "100Mi"},  # 匹配容器实际使用的91Mi，留少量余量
        limits={"ephemeral-storage": "200Mi"}     # 限制最大使用200Mi，防止过度占用
    )
    
    container = client.V1Container(
        name=container_name, 
        image=container_image, 
        env=env_list, 
        stdin=True, 
        security_context=security,
        resources=resources  # 关联资源配置
    )
    template.template.spec = client.V1PodSpec(containers=[container], restart_policy='Never')
    body.spec = client.V1JobSpec(backoff_limit=0, ttl_seconds_after_finished=600, template=template.template)
    return body

参数说明 #

• requests.ephemeral-storage：设置为100Mi是因为你的容器实际用了91Mi，留少量余量避免触发阈值。
• limits.ephemeral-storage：设置为200Mi是防止容器后续运行时占用更多存储，你可以根据实际业务需求调整这个值。

• 用kubectl top pods --containers命令查看容器的实际存储使用量，能更精准地设置资源请求和限制。
• 如果频繁出现节点存储不足，考虑给GKE节点池配置更大的临时存储，或者开启节点池的自动扩容功能。

内容的提问来源于stack exchange，提问作者Turgut