无法从JupyterHub运行Spark作业：Kubernetes连接异常求助

背景 #

已在Minikube上完成Spark部署，本地使用spark-submit能成功运行SparkPi示例，对应Pod状态为Completed。但部署JupyterHub并配置Spark连接时，出现无法连接Pod的错误。

本地成功执行的SparkPi提交命令 #

/opt/spark/bin/spark-submit \
  --master k8s://https://127.0.0.1:62013 \
  --deploy-mode cluster \
  --driver-memory 1g \
  --conf spark.kubernetes.memoryOverheadFactor=0.5 \
  --name sparkpi-test1 \
  --class org.apache.spark.examples.SparkPi \
  --conf spark.kubernetes.container.image=spark:latest \
  --conf spark.kubernetes.driver.pod.name=spark-test1-pi \
  --conf spark.kubernetes.namespace=spark \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
  --verbose \
  local:///opt/spark/examples/jars/spark-examples_2.12-3.3.2.jar 1000

JupyterHub中的Spark配置代码 #

from pyspark import SparkConf, SparkContext
conf = (SparkConf().setMaster("k8s://https://127.0.0.1:52750") # Kubernetes master地址
.set("spark.kubernetes.container.image", "pyspark:latest") # Spark镜像名称
.set("spark.driver.port", "2222") # 需与Service端口匹配
.set("spark.driver.blockManager.port", "7777")
.set("spark.driver.host", "driver-service.jupyterhub.svc.cluster.local") # 需与Service名称匹配
.set("spark.driver.bindAddress", "0.0.0.0")
.set("spark.kubernetes.namespace", "spark")
.set("spark.kubernetes.authenticate.driver.serviceAccountName", "spark")
.set("spark.kubernetes.authenticate.serviceAccountName", "spark")
.set("spark.executor.instances", "2")
.set("spark.kubernetes.container.image.pullPolicy", "IfNotPresent")
.set("spark.app.name", "tutorial_app"))

# 创建SparkContext
sc = SparkContext(conf=conf)

错误信息 #

/home/jovyan/.local/lib/python3.9/site-packages/pyspark/bin/load-spark-env.sh: line 68: ps: command not found
**Error: Caused by: java.net.ConnectException: Failed to connect to /127.0.0.1:52750**

集群资源状态 #

jupyterhub命名空间资源 #

root@TIGER03720:~# k get all -n jupyterhub -o wide
NAME                                  READY   STATUS    RESTARTS      AGE    IP            NODE       NOMINATED NODE   READINESS GATES
pod/continuous-image-puller-qrjr8     1/1     Running   1 (42s ago)   104m   10.244.0.14   minikube   <none>           <none>
pod/hub-6d64d94c89-54vvp              0/1     Error     0             104m   <none>        minikube   <none>           <none>
pod/jupyter-admin                     0/1     Error     0             102m   10.244.0.18   minikube   <none>           <none>
pod/proxy-5c6db96f8-wg9jc             0/1     Running   1 (42s ago)   104m   10.244.0.16   minikube   <none>           <none>
pod/user-scheduler-86cfcff58c-v4m8q   0/1     Running   1 (42s ago)   104m   10.244.0.15   minikube   <none>           <none>
pod/user-scheduler-86cfcff58c-zdv9n   0/1     Running   1 (42s ago)   104m   10.244.0.20   minikube   <none>           <none>

NAME                     TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE    SELECTOR
service/driver-service   ClusterIP      10.110.72.226    <none>        2222/TCP,7777/TCP,4040/TCP   109m   app=jupyterhub,component=singleuser-server
service/hub              ClusterIP      10.106.88.232    <none>        8081/TCP                     104m   app=jupyterhub,component=hub,release=jupyterhub
service/proxy-api        ClusterIP      10.105.217.11    <none>        8001/TCP                     104m   app=jupyterhub,component=proxy,release=jupyterhub
service/proxy-public     LoadBalancer   10.108.107.209   <pending>     80:31888/TCP                 104m   component=proxy,release=jupyterhub

NAME                                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE    CONTAINERS   IMAGES                 SELECTOR
daemonset.apps/continuous-image-puller   1         1         1       1            1           <none>          104m   pause        k8s.gcr.io/pause:3.8   app=jupyterhub,component=continuous-image-puller,release=jupyterhub

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS       IMAGES
                         SELECTOR
deployment.apps/hub              0/1     1            0           104m   hub              jupyterhub/k8s-hub:2.0.0                   app=jupyterhub,component=hub,release=jupyterhub
deployment.apps/proxy            0/1     1            0           104m   chp              jupyterhub/configurable-http-proxy:4.5.3   app=jupyterhub,component=proxy,release=jupyterhub
deployment.apps/user-scheduler   0/2     2            0           104m   kube-scheduler   k8s.gcr.io/kube-scheduler:v1.23.10         app=jupyterhub,component=user-scheduler,release=jupyterhub

NAME                                        DESIRED   CURRENT   READY   AGE    CONTAINERS       IMAGES                                     SELECTOR
replicaset.apps/hub-6d64d94c89              1         1         0       104m   hub              jupyterhub/k8s-hub:2.0.0                   app=jupyterhub,component=hub,pod-template-hash=6d64d94c89,release=jupyterhub
replicaset.apps/proxy-5c6db96f8             1         1         0       104m   chp              jupyterhub/configurable-http-proxy:4.5.3   app=jupyterhub,component=proxy,pod-template-hash=5c6db96f8,release=jupyterhub
replicaset.apps/user-scheduler-86cfcff58c   2         2         0       104m   kube-scheduler   k8s.gcr.io/kube-scheduler:v1.23.10         app=jupyterhub,component=user-scheduler,pod-template-hash=86cfcff58c,release=jupyterhub

spark命名空间资源 #

root@TIGER03720:~# k get all -n spark -o wide
NAME                  READY   STATUS      RESTARTS   AGE    IP           NODE       NOMINATED NODE   READINESS GATES
pod/spark-test1-pi2   0/1     Completed   0          114m   10.244.0.3   minikube   <none>           <none>

解决方案建议 #

• 修正Kubernetes API Server地址：JupyterHub容器内无法访问本地127.0.0.1:52750，需替换为集群内部API Server地址，比如k8s://https://kubernetes.default.svc.cluster.local:443，或从集群kubeconfig中获取正确的内部端点。
• 安装缺失的ps命令：JupyterHub使用的镜像缺少ps工具，可在镜像构建时添加apt-get install -y procps，或在Pod启动时执行安装命令。
• 修复JupyterHub自身部署：从集群状态看，hub和jupyter-admin Pod处于Error状态，先排查并修复JupyterHub的部署问题，确保核心组件正常运行。
• 调整ServiceAccount权限与命名空间：若Spark驱动在jupyterhub命名空间运行，需确保spark ServiceAccount在该命名空间拥有足够权限；或修改Spark配置的spark.kubernetes.namespace为jupyterhub。
• 验证驱动Service配置：确认driver-service的标签选择器与JupyterHub singleuser Pod的标签匹配，确保Service能正确路由到驱动Pod。

内容的提问来源于stack exchange，提问作者KUNAL DAS