Spark提交至K3s集群报错：需显式添加EC Keys依赖到类路径

我正在运行由Rancher Desktop创建的本地K3s Kubernetes集群。kubectl cluster-info返回：

Kubernetes control plane is running at https://127.0.0.1:6443
# ...

执行以下Scala编写的Spark应用提交命令时：

spark-submit \
        --master=k8s://https://127.0.0.1:6443 \
        --deploy-mode=cluster \
        --name=findretiredpeople \
        --class=com.sundogsoftware.spark.Hello \
        --conf=spark.executor.instances=2 \
        --conf=spark.kubernetes.container.image=hongbo-miao/hello:latest \
        local:///target/scala-2.12/hello_2.12-1.0.jar

触发如下错误：

23/03/20 18:14:10 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
23/03/20 18:14:10 INFO SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file
Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: JcaPEMKeyConverter is provided by BouncyCastle, an optional dependency. To use support for EC Keys you must explicitly add this dependency to classpath.
  at io.fabric8.kubernetes.client.internal.CertUtils.handleECKey(CertUtils.java:164)
  at io.fabric8.kubernetes.client.internal.CertUtils.loadKey(CertUtils.java:134)
  at io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:112)
  at io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:247)
  at io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:153)
  at io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:147)
  at io.fabric8.kubernetes.client.utils.HttpClientUtils.applyCommonConfiguration(HttpClientUtils.java:204)
  at io.fabric8.kubernetes.client.okhttp.OkHttpClientFactory.createHttpClient(OkHttpClientFactory.java:89)
  at org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:118)
  at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$4(KubernetesClientApplication.scala:242)
  at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2763)
  at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:242)
  at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:214)
  at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
  at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
  at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
  at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
  at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
  at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
  at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
23/03/20 18:14:10 INFO ShutdownHookManager: Shutdown hook called
23/03/20 18:14:10 INFO ShutdownHookManager: Deleting directory /private/var/folders/22/ntjwd5dx691gvkktkspl0f_00000gq/T/spark-15d1055d-c2fb-40d4-81ff-66b602595979

错误核心是Spark的Kubernetes客户端需要BouncyCastle库处理EC（椭圆曲线）密钥，但该库默认不在Spark类路径中，以下是三种解决方式：

1. 提交命令中直接引入依赖 #

通过--packages参数从Maven仓库拉取BouncyCastle包，修改后的提交命令如下：

spark-submit \
        --master=k8s://https://127.0.0.1:6443 \
        --deploy-mode=cluster \
        --name=findretiredpeople \
        --class=com.sundogsoftware.spark.Hello \
        --conf=spark.executor.instances=2 \
        --conf=spark.kubernetes.container.image=hongbo-miao/hello:latest \
        --packages org.bouncycastle:bcprov-jdk15on:1.70 \
        local:///target/scala-2.12/hello_2.12-1.0.jar

2. 自定义Spark镜像内置依赖 #

如果使用自定义Spark镜像，可在Dockerfile中添加BouncyCastle jar包到Spark的jars目录：

FROM your-base-spark-image
RUN curl -o $SPARK_HOME/jars/bcprov-jdk15on-1.70.jar https://repo1.maven.org/maven2/org/bouncycastle/bcprov-jdk15on/1.70/bcprov-jdk15on-1.70.jar

3. 调整K3s密钥类型（可选） #

若不需要EC密钥，可修改K3s启动参数，改用RSA密钥。添加启动参数--tls-san 127.0.0.1 --kube-apiserver-arg=service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key（路径根据实际部署调整），之后重启K3s服务。此方法复杂度较高，优先推荐前两种。

内容的提问来源于stack exchange，提问作者Hongbo Miao