Dataproc Batch运行Scala/Spark调用SecretManagerClient.create()遇NoSuchMethodError

安全阿岳

2026-6-18

问题：Dataproc上调用Google Secret Manager客户端出现NoSuchMethodError错误

场景说明

我尝试从Google Secret Manager读取密钥，通过SBT构建Scala/Spark Fat JAR并部署在Dataproc上。调用以下代码时：

val client: SecretManagerServiceClient = SecretManagerServiceClient.create()

在Dataproc上触发错误。

错误日志

Using the default container image
Waiting for container log creation
PYSPARK_PYTHON=/opt/dataproc/conda/bin/python
Generating /home/spark/.pip/pip.conf
Configuring index-url as 'https://europe-python.pkg.dev/artifact-registry-python-cache/virtual-python/simple/'
JAVA_HOME=/usr/lib/jvm/temurin-17-jdk-amd64
SPARK_EXTRA_CLASSPATH=
:: loading settings :: file = /etc/spark/conf/ivysettings.xml
24/08/30 13:35:09 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-google-hadoop-file-system.properties,hadoop-metrics2.properties
24/08/30 13:35:09 INFO MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
24/08/30 13:35:09 INFO MetricsSystemImpl: google-hadoop-file-system metrics system started
Exception in thread "main" java.lang.NoSuchMethodError: 'io.grpc.MethodDescriptor$Marshaller io.grpc.protobuf.ProtoUtils.marshaller(repackaged.com.google.protobuf.Message)'
        at com.google.cloud.secretmanager.v1.stub.GrpcSecretManagerServiceStub.&lt;clinit&gt;(GrpcSecretManagerServiceStub.java:72)
        at com.google.cloud.secretmanager.v1.stub.SecretManagerServiceStubSettings.createStub(SecretManagerServiceStubSettings.java:350)
        at com.google.cloud.secretmanager.v1.SecretManagerServiceClient.&lt;init&gt;(SecretManagerServiceClient.java:455)
        at com.google.cloud.secretmanager.v1.SecretManagerServiceClient.create(SecretManagerServiceClient.java:437)
        at com.google.cloud.secretmanager.v1.SecretManagerServiceClient.create(SecretManagerServiceClient.java:428)
        at fr.mycomp.graphuser.GraphUserApp$.getCredentials(GraphUserApp.scala:30)
        at fr.mycomp.graphuser.GraphUserApp$.delayedEndpoint$fr$mycomp$graphuser$GraphUserApp$1(GraphUserApp.scala:95)
        at fr.mycomp.graphuser.GraphUserApp$delayedInit$body.apply(GraphUserApp.scala:15)
        at scala.Function0.apply$mcV$sp(Function0.scala:39)
        at scala.Function0.apply$mcV$sp$(Function0.scala:39)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
        at scala.App.$anonfun$main$1$adapted(App.scala:80)
        at scala.collection.immutable.List.foreach(List.scala:431)
        at scala.App.main(App.scala:80)
        at scala.App.main$(App.scala:78)
        at fr.mycomp.graphuser.GraphUserApp$.main(GraphUserApp.scala:15)
        at fr.mycomp.graphuser.GraphUserApp.main(GraphUserApp.scala)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1032)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1124)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1133)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
ERROR: (gcloud.dataproc.batches.submit.spark) Batch job is FAILED. Detail: Job failed with message [Exception in thread "main" java.lang.NoSuchMethodError: 'io.grpc.MethodDescriptor$Marshaller io.grpc.protobuf.ProtoUtils.marshaller(repackaged.com.google.protobuf.Message)']

我的build.sbt配置

val sparkVersion = settingKey[String]("Spark version")

lazy val root = (project in file("."))
  .settings(
    inThisBuild(List(
      organization := "fr.mycomp",
      scalaVersion := "2.12.13"
    )),
    name := "graphUser",
    version := "0.0.1",

    sparkVersion := "3.5.0",

    javacOptions ++= Seq("-source", "1.8", "-target", "1.8"),
    javaOptions ++= Seq("-Xms512M", "-Xmx2048M"),
    scalacOptions ++= Seq("-deprecation", "-unchecked"),
    parallelExecution in Test := false,
    fork := true,

    coverageHighlighting := true,

    libraryDependencies ++= Seq(
      "org.apache.spark" %% "spark-core" % sparkVersion.value exclude("com.google.protobuf","protobuf-java"),
      "org.apache.spark" %% "spark-sql" % sparkVersion.value exclude("com.google.protobuf","protobuf-java"),
      "org.apache.spark" %% "spark-graphx" % sparkVersion.value exclude("com.google.protobuf","protobuf-java"),

      // Snowflake Connector for Spark
      "net.snowflake" % "spark-snowflake_2.12" % "3.0.0",
      "net.snowflake" % "snowflake-jdbc" % "3.17.0",

      // https://storage.googleapis.com/cloud-opensource-java-dashboard/com.google.cloud/libraries-bom/26.45.0/index.html
      "com.google.cloud" % "google-cloud-storage" % "2.40.1",
      "com.google.cloud" % "google-cloud-secretmanager" % "2.48.0",
      "com.google.protobuf" % "protobuf-java" % "3.25.4",
      "com.google.protobuf" % "protobuf-java-util" % "3.25.4",
      "io.grpc" % "grpc-all" % "1.66.0",
      "io.grpc" % "grpc-protobuf" % "1.66.0",
      "io.grpc" % "grpc-okhttp" % "1.66.0",
      "io.grpc" % "grpc-protobuf-lite" % "1.66.0",
      "io.grpc" % "grpc-stub" % "1.66.0",

      // PureConfig for configuration management
      "com.github.pureconfig" %% "pureconfig" % "0.17.6",

      // SLF4J logging dependencies
      "org.slf4j" % "slf4j-api" % "2.0.9",
      "org.slf4j" % "slf4j-log4j12" % "2.0.9",

      // Log4J logging dependencies
      "org.apache.logging.log4j" % "log4j-api" % "2.23.1",
      "org.apache.logging.log4j" % "log4j-core" % "2.23.1",

      // JSON4S for JSON parsing
      "org.json4s" %% "json4s-native" % "3.6.6",
      "org.json4s" %% "json4s-jackson" % "3.6.6",
      "org.json4s" %% "json4s-core" % "3.6.6",
      "org.json4s" %% "json4s-ast" % "3.6.6",
      "org.json4s" %% "json4s-scalap" % "3.6.6",

      // Testing dependencies
      "org.scalatest" %% "scalatest" % "3.2.19" % Test,
      "org.scalacheck" %% "scalacheck" % "1.18.0" % Test,
      "com.holdenkarau" %% "spark-testing-base" % s"${sparkVersion.value}_1.5.3" % Test
    ),

    // Configure the run task
    run in Compile := Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run)).evaluated,

    // Additional build settings
    scalacOptions ++= Seq("-deprecation", "-unchecked"),
    pomIncludeRepository := { x => false },

    // Repositories for dependencies
    resolvers ++= Seq(
      "sonatype-releases" at "https://oss.sonatype.org/content/repositories/releases/",
      "Typesafe repository" at "https://repo.typesafe.com/typesafe/releases/",
      "Second Typesafe repo" at "https://repo.typesafe.com/typesafe/maven-releases/",
      "Spark Packages Repo" at "https://repos.spark-packages.org/",
      "Maven Central" at "https://repo1.maven.org/maven2/",
      Resolver.sonatypeRepo("public")
    ),

    // Publishing settings
    publishTo := {
      val nexus = "https://oss.sonatype.org/"
      if (isSnapshot.value)
        Some("snapshots" at nexus + "content/repositories/snapshots")
      else
        Some("releases"  at nexus + "service/local/staging/deploy/maven2")
    }
  )

import sbtassembly.AssemblyPlugin.autoImport._

// Strategy for handling conflicts in META-INF during assembly
assemblyMergeStrategy in assembly := {
  case PathList("META-INF", xs @ _*) => MergeStrategy.discard
  case x => MergeStrategy.first
}

// Shade rules to prevent conflicts in shaded libraries
assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.google.common.**" -> "repackaged.com.google.common.@1").inAll,
  ShadeRule.rename("com.google.protobuf.**" -> "repackaged.com.google.protobuf.@1").inAll,
//  ShadeRule.rename("io.grpc.**" -> "repackaged.io.grpc.@1").inAll
)

启动命令

gcloud dataproc batches submit spark --batch=graphuser-$(date '+%Y%m%d')-$(date +%s) --region=europe-west4 --subnet=dataproc-subnet-eu-west4 --version=1.2 --properties=spark.dataproc.scaling.version=2,spark.dynamicAllocation.enabled=true,spark.dynamicAllocation.initialExecutors=4,spark.dynamicAllocation.minExecutors=4,spark.sql.shuffle.partitions=500,spark.executor.memory=12g,spark.dynamicAllocation.executorAllocationRatio=0.5,spark.dynamicAllocation.maxExecutors=100,spark.driver.memory=9g --ttl=12h --jar=gs://dataproc-data-eng/user_ids/graphUser-assembly-0.0.1.jar

已尝试的方案

遵循Google BOM管理依赖
切换到Maven构建
但问题仍未解决。

解决方案

这个错误的核心是grpc-protobuf库与被shade的protobuf类不兼容：已经将protobuf包重命名为repackaged.com.google.protobuf，但grpc-protobuf的ProtoUtils.marshaller方法仍在寻找原生的com.google.protobuf.Message类型，而非shaded后的版本。

具体修复步骤：

启用grpc的shade规则
取消注释build.sbt中被注释的grpc shade规则，让grpc相关类也使用repackaged命名空间，确保它能识别shaded后的protobuf类：
```
ShadeRule.rename("io.grpc.**" -> "repackaged.io.grpc.@1").inAll
```

调整依赖管理，使用Google Cloud BOM统一版本
直接引入Google Cloud BOM来管理所有GCP相关依赖的版本，避免版本冲突：

libraryDependencies ++= Seq(
  // 引入Google Cloud BOM
  "com.google.cloud" % "libraries-bom" % "26.45.0" % PomOnly,
  // 无需指定版本，由BOM自动管理
  "com.google.cloud" % "google-cloud-storage",
  "com.google.cloud" % "google-cloud-secretmanager",
  // 其他依赖...
)

排除Spark中的grpc和protobuf依赖
除了排除Spark的protobuf，还要排除Spark自带的grpc相关依赖，避免与引入的版本冲突：

"org.apache.spark" %% "spark-core" % sparkVersion.value 
  exclude("com.google.protobuf","protobuf-java")
  exclude("io.grpc", "grpc-netty")
  exclude("io.grpc", "grpc-core")
  exclude("io.grpc", "grpc-stub"),
// spark-sql和spark-graphx做同样的排除

简化grpc依赖
移除grpc-all，只保留必要的grpc依赖，避免重复：

"io.grpc" % "grpc-protobuf" % "1.66.0",
"io.grpc" % "grpc-okhttp" % "1.66.0",
"io.grpc" % "grpc-stub" % "1.66.0"

验证assembly后的依赖
使用sbt dependencyTree命令检查依赖树，确保没有冲突的protobuf或grpc版本被引入。

额外建议

Dataproc环境自带部分GCP客户端库，如果不需要在Fat JAR中打包这些库，可以将GCP依赖标记为provided，避免冲突：
```
"com.google.cloud" % "google-cloud-secretmanager" % "2.48.0" % Provided
```
但需确保Dataproc版本自带的库版本与代码兼容。

内容的提问来源于stack exchange，提问作者Arno

Dataproc Batch运行Scala/Spark调用SecretManagerClient.create()遇NoSuchMethodError

场景说明 #

错误日志 #

我的build.sbt配置 #

启动命令 #

已尝试的方案 #

解决方案 #

具体修复步骤： #

额外建议 #