Spark作业遇Guava版本冲突报错NoSuchMethodError，参数值影响执行结果

环境信息 #

• Hadoop: 3.1.4
• Spark: 3.3.1
• Hive: 3.1.3

问题现象 #

执行SparkPi示例作业时，不带参数或传入参数小于50会抛出java.lang.NoSuchMethodError错误；但当命令末尾传入的参数≥50时，作业可成功执行。

报错详情 #

报错命令 #

/opt/spark-3.3.1/bin/spark-submit --class org.apache.spark.examples.SparkPi --deploy-mode cluster --master yarn --num-executors 1 --driver-memory 1G --executor-memory 1G --executor-cores 1  /opt/spark-3.3.1/examples/jars/spark-examples_2.12-3.3.1.jar

报错日志 #

[2022-12-09 00:05:02.747]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data/hdfsdata2/yarn/local/usercache/spark/filecache/70/__spark_libs__3692263374412677830.zip/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-3.1.4/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2022-12-09 00:05:02,137 INFO util.SignalUtils: Registering signal handler for TERM
2022-12-09 00:05:02,139 INFO util.SignalUtils: Registering signal handler for HUP
2022-12-09 00:05:02,139 INFO util.SignalUtils: Registering signal handler for INT
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
    at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
    at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
    at org.apache.spark.deploy.SparkHadoopUtil$.$anonfun$appendHiveConfigs$1(SparkHadoopUtil.scala:477)
    at org.apache.spark.deploy.SparkHadoopUtil$.$anonfun$appendHiveConfigs$1$adapted(SparkHadoopUtil.scala:476)
    at scala.collection.immutable.Stream.foreach(Stream.scala:533)
    at org.apache.spark.deploy.SparkHadoopUtil$.appendHiveConfigs(SparkHadoopUtil.scala:476)
    at org.apache.spark.deploy.SparkHadoopUtil$.org$apache$spark$deploy$SparkHadoopUtil$$appendS3AndSparkHadoopHiveConfigurations(SparkHadoopUtil.scala:456)
    at org.apache.spark.deploy.SparkHadoopUtil$.newConfiguration(SparkHadoopUtil.scala:430)
    at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:894)
    at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)

已尝试操作 #

经调试确认问题与Guava版本冲突有关：Hadoop使用guava-27.0-jre.jar，Spark默认使用guava-14.0-jre.jar。已将Spark的Guava及相关依赖jar替换为Hadoop对应版本，替换的jar列表如下：

• /opt/spark-3.3.1/jars/animal-sniffer-annotations-1.17.jar
• /opt/spark-3.3.1/jars/failureaccess-1.0.jar
• /opt/spark-3.3.1/jars/error_prone_annotations-2.2.0.jar
• /opt/spark-3.3.1/jars/checker-qual-2.5.2.jar
• /opt/spark-3.3.1/jars/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
• /opt/spark-3.3.1/jars/jsr305-3.0.2.jar
• /opt/spark-3.3.1/jars/j2objc-annotations-1.1.jar
• /opt/spark-3.3.1/jars/guava-27.0-jre.jar

替换后问题仍未解决。

成功执行的命令 #

/opt/spark-3.3.1/bin/spark-submit --class org.apache.spark.examples.SparkPi --deploy-mode cluster --master yarn --num-executors 1 --driver-memory 1G --executor-memory 1G --executor-cores 1  /opt/spark-3.3.1/examples/jars/spark-examples_2.12-3.3.1.jar 50

求助点 #

不清楚为何参数大小会影响作业执行结果，请求解决该Guava版本冲突问题的有效方案。

内容的提问来源于stack exchange，提问作者Saurav Suman