Spark on AKS驱动Pod启动正常但无法创建Executor求助

环境配置 #

• 子网中的AKS集群，采用托管控制平面
• 已完成操作：
  1. 创建命名空间
  2. 创建带令牌的ServiceAccount、角色及角色绑定
  3. 拉取该ServiceAccount的kubeconfig文件

执行的Spark提交命令 #

./bin/spark-submit
--master k8s://https://AKS-API-URL:443
--deploy-mode cluster
--name spark-pi
--conf spark.kubernetes.namespace=MYNAMESPACE
--conf spark.kubernetes.authenticate.driver.serviceAccountName=SA-ACCCOUNT
--conf spark.kubernetes.authenticate.executor.serviceAccountName=SA-ACCCOUNT
--class org.apache.spark.examples.SparkPi
--conf spark.executor.instances=2
--conf spark.kubernetes.container.image=spark3.2.3-h3.2.2:03092023
--conf spark.kubernetes.driver.pod.name=spark-pi
--conf spark.kubernetes.driver.limit.cores=250m
--conf spark.kubernetes.executor.limit.cores=500m
--conf spark.kubernetes.executor.request.cores=250m
--conf spark.kubernetes.driver.request.cores=250m
--conf "spark.driver.extraJavaOptions=-Dlog4j.logger.org.apache.spark=DEBUG"
local:///opt/oss/spark/examples/jars/spark-examples_2.12-3.2.3.jar

报错信息 #

ERROR spark.SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: External scheduler cannot be instantiated
at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2979)
at org.apache.spark.SparkContext.(SparkContext.scala:559)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:30)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get] for kind: [Pod] with name: [spark-pi] in namespace: [MYNAMESPACE] failed.
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:226)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:187)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:86)
at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$driverPod$1(ExecutorPodsAllocator.scala:79)
at scala.Option.map(Option.scala:230)
at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.(ExecutorPodsAllocator.scala:78)
at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:118)
at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2973)
... 19 more
Caused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1696)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1514)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1416)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:427)
at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:320)
at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:284)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:169)
at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:258)
at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:133)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.intercept(TokenRefreshInterceptor.java:42)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at io.fabric8.kubernetes.client.utils.HttpClientUtils.lambda$createApplicableInterceptors$6(HttpClientUtils.java:284)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
at okhttp3.RealCall.execute(RealCall.java:93)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:541)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:504)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:471)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:453)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:947)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:221)
... 26 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:483)
at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472)
at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506)
... 66 more

用户疑问 #

驱动Pod能正常启动，但无法创建Executor，是否存在遗漏配置？该问题属于RBAC权限问题还是网络问题？请提供排查思路。

从报错堆栈看，核心是SSL握手失败导致Driver无法访问K8s API Server获取自身Pod信息，进而无法初始化调度器创建Executor。按以下步骤逐步排查：

一、优先排查网络/SSL问题 #

1. 进入Driver Pod测试网络连通性
   • 执行kubectl exec -it spark-pi -n MYNAMESPACE -- /bin/bash进入Driver容器
   • 用curl测试API Server连通性：curl -v https://AKS-API-URL:443/api/v1/namespaces/MYNAMESPACE/pods/spark-pi
     • 若连不通，说明Driver Pod到API Server的网络路径被拦截，检查子网NSG规则是否允许443端口访问AKS API Server的IP段，或AKS API Server是否设置了授权IP范围（需将Driver所在子网的出口IP加入列表）
     • 若出现SSL错误，验证容器内是否信任AKS的CA证书：
       • 执行curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt https://AKS-API-URL:443/api/v1/namespaces/MYNAMESPACE/pods/spark-pi
2. 确认API Server地址正确性
   • 通过kubectl cluster-info获取集群实际API地址，对比提交命令中的AKS-API-URL是否一致

二、排除网络后验证RBAC权限 #

虽然报错是SSL问题，但也可能是权限不足导致API Server主动断开连接，需验证：

1. 直接用ServiceAccount令牌测试API访问
   • 获取令牌：kubectl get secret $(kubectl get sa SA-ACCCOUNT -n MYNAMESPACE -o jsonpath='{.secrets[0].name}') -n MYNAMESPACE -o jsonpath='{.data.token}' | base64 -d
   • 执行curl -H "Authorization: Bearer $TOKEN" --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt https://AKS-API-URL:443/api/v1/namespaces/MYNAMESPACE/pods/spark-pi
     • 返回403则说明ServiceAccount缺少get pods权限，需更新Role/ClusterRole的规则
2. 快速验证权限
   • 执行kubectl auth can-i get pods -n MYNAMESPACE --as=system:serviceaccount:MYNAMESPACE:SA-ACCCOUNT，确认返回yes

三、检查Spark配置遗漏项 #

1. 指定CA证书路径
   • 若Spark镜像未默认加载K8s CA证书，添加配置：--conf spark.kubernetes.authenticate.caCertFile=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
2. 验证ServiceAccount挂载状态
   • 执行kubectl describe pod spark-pi -n MYNAMESPACE，查看Volumes和VolumeMounts中是否包含kube-api-access-xxx类型的Secret，确保token和ca.crt已正确挂载

内容的提问来源于stack exchange，提问作者Bhaskar