Spark Kubernetes报错：Pod已存在（Zeppelin Livy提交场景）

错误信息 #

Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://kubernetes.default.svc.cluster.local/api/v1/namespaces/mynamespace/pods. Message: pods "livy-session-71-aa59b084378baed1-driver" already exists. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null, kind=pods, name=livy-session-71-aa59b084378baed1-driver, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=pods "livy-session-71-aa59b084378baed1-driver" already exists, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=AlreadyExists, status=Failure, additionalProperties={}).
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:251)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:815)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:333)
    at org.apache.spark.deploy.k8s.submit.Client.$anonfun$run$2(KubernetesClientApplication.scala:130)
    at org.apache.spark.deploy.k8s.submit.Client.$anonfun$run$2$adapted(KubernetesClientApplication.scala:129)
    at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2539)
    at org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:129)
    at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$4(KubernetesClientApplication.scala:221)
    at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$4$adapted(KubernetesClientApplication.scala:215)
    at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2539)
    at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:215)
    at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:188)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

解决方法 #

• 强制清理K8s残留资源记录
  尽管Pod不存在，但K8s API Server可能存在缓存或残留的资源元数据。执行以下命令强制清理目标Pod记录：

  kubectl delete pod livy-session-71-aa59b084378baed1-driver --namespace mynamespace --force --grace-period=0
  

  同时检查是否有与该会话关联的PVC、ConfigMap等资源，如有残留也一并删除。

• 重启Livy服务
  Livy可能缓存了旧会话的Pod名称信息，导致重复使用已失效的名称。根据你的部署方式重启Livy：

  # 若Livy以Deployment部署
  kubectl rollout restart deployment livy --namespace mynamespace
  

• 修改Spark Driver Pod命名策略
  在Zeppelin的Livy解释器配置中添加随机化命名参数，确保每次创建的Driver Pod名称唯一：

  spark.kubernetes.driver.name=livy-session-${livy.session.id}-${random.uuid}-driver
  

• 检查K8s API Server状态
  若API Server存在缓存不一致问题，等待数分钟让集群状态同步；若问题持续，检查API Server日志排查异常，必要时在集群高可用前提下重启API Server组件。

• 清理Zeppelin会话缓存
  在Zeppelin界面中强制终止并删除关联的Livy会话，重新创建会话后提交Spark任务，避免复用旧的会话配置。

内容的提问来源于stack exchange，提问作者Arunav Jyoti Ojah