Kubernetes Spark Operator Driver Pod报NoSuchMethodError问题求助

我们在安装Kubernetes Spark Operator并编写连接S3写入文件的SparkApplication示例时，始终遇到以下错误：

WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.1.2.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
23/01/15 15:00:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.NoSuchMethodError: 'char[] org.apache.hadoop.conf.Configuration.getPassword(java.lang.String)'
        at org.apache.spark.SSLOptions$.$anonfun$parse$8(SSLOptions.scala:188)
        at scala.Option.orElse(Option.scala:447)
        at org.apache.spark.SSLOptions$.parse(SSLOptions.scala:188)
        at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:98)
        at org.apache.spark.deploy.SparkSubmit.secMgr$lzycompute$1(SparkSubmit.scala:368)
        at org.apache.spark.deploy.SparkSubmit.secMgr$1(SparkSubmit.scala:368)
        at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$8(SparkSubmit.scala:376)
        at scala.Option.map(Option.scala:230)
        at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:376)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

我们的Spark应用创建流程如下：

1. 构建Spark基础镜像 #

基于spark-3.1.1-bin-hadoop3.2包构建基础镜像：

$ cd spark-3.1.1-bin-hadoop3.2
$ ./bin/docker-image-tool.sh -r <registryurl>/nks/sparkoperator/base -t 3.1.1 -u 1000 -b java_image_tag=11-jre-slim build 

生成的镜像地址：<registryurl>/nks/sparkoperator/base/spark:3.1.1

2. 构建应用镜像 #

应用文件结构：

.
├── Dockerfile
├── build.sbt
├── plugins.sbt
└── src
    └── main
        └── scala
            └── com
                └── company
                    └── xyz
                        └── ParquetAWSExample.scala

Dockerfile内容： #

FROM <registryurl>/nks/sparkoperator/base/spark:3.1.1
USER root

RUN apt -y install wget

ARG SBT_VERSION
ENV SBT_VERSION=${SBT_VERSION:-1.5.1}

RUN wget -O - https://github.com/sbt/sbt/releases/download/v${SBT_VERSION}/sbt-${SBT_VERSION}.tgz | gunzip | tar -x -C /usr/local

ENV PATH /usr/local/sbt/bin:${PATH}

WORKDIR /app
COPY . /app
ADD plugins.sbt /app/project/
RUN sbt update

RUN sbt clean assembly

构建出的镜像：<registryurl>/nks/testsparkoperatorv2/s3conn:1.5

build.sbt内容： #

name := "xyz"
version := "0.1"

scalaVersion := "2.12.11"


libraryDependencies ++= Seq(
      "org.apache.spark" %% "spark-core" % "3.1.1" ,
      "org.apache.spark" %% "spark-sql" % "3.1.1",
      "org.apache.hadoop" % "hadoop-aws" % "3.1.1",
    )

dependencyOverrides += "org.apache.hadoop" % "hadoop-common" % "3.1.1"

assemblyMergeStrategy in assembly := {
  case PathList("META-INF", xs @ _*) => MergeStrategy.discard
  case x => MergeStrategy.first
}

spark-application.yaml内容： #

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: parquet.test
  namespace: spark-operator
spec:
  type: Scala
  mode: cluster
  image: "<registryurl>/nks/testsparkoperatorv2/s3conn:1.5"
  imagePullPolicy: Always
  imagePullSecrets:
    - myregistrykey
  mainClass: com.company.xyz.ParquetAWSExample
  mainApplicationFile: "local:///app/target/scala-2.12/xyz-assembly-0.1.jar"
  sparkVersion: "3.1.1"
  driver:
    memory: 512m
    labels:
      version: 3.1.1
    serviceAccount: sparkoperator
    volumeMounts:
      - name: "test-volume"
        mountPath: "/tmp"
  executor:
    cores: 1
    instances: 1
    memory: 512m
    labels:
      version: 3.1.1
    volumeMounts:
      - name: "test-volume"
        mountPath: "/tmp"

错误原因 #

核心问题是Hadoop版本不匹配：

• 基础镜像基于spark-3.1.1-bin-hadoop3.2构建，内置Hadoop版本为3.2.x
• 但build.sbt中强制指定hadoop-common和hadoop-aws为3.1.1，运行时出现类方法签名不兼容（getPassword方法在Hadoop 3.2中存在，3.1.1中无对应签名）

修复步骤 #

1. 统一Hadoop版本 #

修改build.sbt，将Hadoop相关依赖版本改为3.2.x，与基础镜像版本保持一致，同时给Spark核心依赖添加Provided避免重复打包冲突：

name := "xyz"
version := "0.1"

scalaVersion := "2.12.11"

libraryDependencies ++= Seq(
      "org.apache.spark" %% "spark-core" % "3.1.1" % Provided,
      "org.apache.spark" %% "spark-sql" % "3.1.1" % Provided,
      "org.apache.hadoop" % "hadoop-aws" % "3.2.0",
    )

dependencyOverrides ++= Seq(
  "org.apache.hadoop" % "hadoop-common" % "3.2.0",
  "org.apache.hadoop" % "hadoop-client" % "3.2.0"
)

assemblyMergeStrategy in assembly := {
  case PathList("META-INF", xs @ _*) => MergeStrategy.discard
  case x => MergeStrategy.first
}

2. 重新构建应用镜像 #

删除旧镜像，重新执行构建并推送：

docker build -t <registryurl>/nks/testsparkoperatorv2/s3conn:1.6 .
docker push <registryurl>/nks/testsparkoperatorv2/s3conn:1.6

3. 更新SparkApplication配置 #

修改spark-application.yaml中的镜像版本为新构建的1.6：

image: "<registryurl>/nks/testsparkoperatorv2/s3conn:1.6"

4. 重新提交应用 #

删除旧应用并重新部署：

kubectl delete sparkapplication parquet.test -n spark-operator
kubectl apply -f spark-application.yaml

额外注意事项 #

• 确保Spark与Hadoop版本兼容：Spark 3.1.1官方推荐搭配Hadoop 3.2.x，避免跨版本混用
• 连接S3需添加相关配置到spark-application.yaml的spec.sparkConf中，敏感密钥建议通过K8s Secret管理，示例：

spec:
  ...
  sparkConf:
    "spark.hadoop.fs.s3a.access.key": "你的访问密钥"
    "spark.hadoop.fs.s3a.secret.key": "你的私有密钥"
    "spark.hadoop.fs.s3a.endpoint": "s3.amazonaws.com"
    "spark.hadoop.fs.s3a.impl": "org.apache.hadoop.fs.s3a.S3AFileSystem"

内容的提问来源于stack exchange，提问作者Pro