Windows10下初始化Spark实例报错：无法识别Hadoop主版本号

API 探路者

2026-8-2

问题描述

我在Windows 10系统上初始化Apache Spark实例做本地测试时遇到错误，之前代码一直运行正常，推测是依赖或配置变更导致的。当前环境：

JDK版本：1.8.0_192（试过换成1.8.0_181和jdk11+28-x64，无效）
Hadoop版本：3.0.0（HADOOP_HOME设为C:\Data\devtools\hadoop-win\3.0.0，删除过该环境变量也无效）
Spark版本：2.4.0
构建工具：Maven

SparkSession初始化代码

def withSparkSession(testMethod: SparkSession => Any) {

  val uuid = UUID.randomUUID().toString

  val pathRoot = s"C:/data/temp/spark-testcase/$uuid" // TODO: make this independent from Windows
  val derbyRoot = s"C:/data/temp/spark-testcase/derby_system_root"

  // TODO: clear me up -- Derby based metastore should be cleared up
  System.setProperty("derby.system.home", s"${derbyRoot}")

  val conf = new SparkConf()
    .set("testcase.root.dir", s"${pathRoot}")
    .set("spark.sql.warehouse.dir", s"${pathRoot}/test-hive-dwh")
    .set("spark.sql.catalogImplementation", "hive")
    .set("hive.exec.scratchdir", s"${pathRoot}/hive-scratchdir")
    .set("hive.exec.dynamic.partition.mode", "nonstrict")
    .setMaster("local[*]")
    .setAppName("Spark Hive Test case")

  val spark = SparkSession.builder()
    .config(conf)
    .enableHiveSupport()
    .getOrCreate()

  try {
    testMethod(spark)
  }
  finally {
    spark.sparkContext.stop()
    println(s"Deleting test case root directory: $pathRoot")
    deleteRecursively(nioPaths.get(pathRoot))
  }
}

错误日志

An exception or error caused a run to abort. 
java.lang.ExceptionInInitializerError
    at org.apache.hadoop.hive.conf.HiveConf.<clinit>(HiveConf.java:105)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.apache.spark.util.Utils$.classForName(Utils.scala:238)
    at org.apache.spark.sql.SparkSession$.hiveClassesArePresent(SparkSession.scala:1117)
    at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:866)
.
.
.
    at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
    at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
    at org.scalatest.Transformer.apply(Transformer.scala:22)
    at org.scalatest.Transformer.apply(Transformer.scala:20)
    at org.scalatest.FunSpecLike$$anon$1.apply(FunSpecLike.scala:454)
    at org.scalatest.TestSuite$class.withFixture(TestSuite.scala:196)
    at org.scalamock.scalatest.AbstractMockFactory$$anonfun$withFixture$1.apply(AbstractMockFactory.scala:35)
    at org.scalamock.scalatest.AbstractMockFactory$$anonfun$withFixture$1.apply(AbstractMockFactory.scala:34)
    at org.scalamock.MockFactoryBase$class.withExpectations(MockFactoryBase.scala:41)
    at org.scalamock.scalatest.AbstractMockFactory$class.withFixture(AbstractMockFactory.scala:34)
    at org.scalatest.FunSpecLike$class.invokeWithFixture$1(FunSpecLike.scala:451)
    at org.scalatest.FunSpecLike$$anonfun$runTest$1.apply(FunSpecLike.scala:464)
    at org.scalatest.FunSpecLike$$anonfun$runTest$1.apply(FunSpecLike.scala:464)
    at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
    at org.scalatest.FunSpecLike$class.runTest(FunSpecLike.scala:464)
    at org.scalatest.FunSpec.runTest(FunSpec.scala:1630)
    at org.scalatest.FunSpecLike$$anonfun$runTests$1.apply(FunSpecLike.scala:497)
    at org.scalatest.FunSpecLike$$anonfun$runTests$1.apply(FunSpecLike.scala:497)
    at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:396)
    at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384)
    at scala.collection.immutable.List.foreach(List.scala:392)
    at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
    at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:373)
    at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:410)
    at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384)
    at scala.collection.immutable.List.foreach(List.scala:392)
    at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
    at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:379)
    at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
    at org.scalatest.FunSpecLike$class.runTests(FunSpecLike.scala:497)
    at org.scalatest.FunSpec.runTests(FunSpec.scala:1630)
    at org.scalatest.Suite$class.run(Suite.scala:1147)
    at org.scalatest.FunSpec.org$scalatest$FunSpecLike$$super$run(FunSpec.scala:1630)
    at org.scalatest.FunSpecLike$$anonfun$run$1.apply(FunSpecLike.scala:501)
    at org.scalatest.FunSpecLike$$anonfun$run$1.apply(FunSpecLike.scala:501)
    at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
    at org.scalatest.FunSpecLike$class.run(FunSpecLike.scala:501)
    at org.scalatest.FunSpec.run(FunSpec.scala:1630)
    at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:45)
    at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$1.apply(Runner.scala:1346)
    at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$1.apply(Runner.scala:1340)
    at scala.collection.immutable.List.foreach(List.scala:392)
    at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1340)
    at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1011)
    at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1010)
    at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1506)
    at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1010)
    at org.scalatest.tools.Runner$.run(Runner.scala:850)
    at org.scalatest.tools.Runner.run(Runner.scala)
    at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2or3(ScalaTestRunner.java:43)
    at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:26)
Caused by: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.0.0-cdh6.3.4
    at org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion(ShimLoader.java:174)
    at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:139)
    at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:100)
    at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:368)
    ... 64 more


Process finished with exit code 2

解决方案

从错误根因Unrecognized Hadoop major version number: 3.0.0-cdh6.3.4来看，问题出在Hive的ShimLoader无法识别带CDH后缀的Hadoop版本号。Spark 2.4.0依赖的Hive版本（通常是1.2.1）对Hadoop 3.x的支持有限，尤其是CDH定制版本。

以下是可行的解决步骤：

调整Maven依赖，强制指定兼容的Hadoop版本
在pom.xml中添加Hadoop相关依赖，排除默认的CDH版本，使用官方3.0.0版本：

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>3.0.0</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-common</artifactId>
    <version>3.0.0</version>
    <scope>provided</scope>
</dependency>

如果项目中引入了CDH相关依赖，需显式排除：

<exclusions>
    <exclusion>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
    </exclusion>
    <exclusion>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-common</artifactId>
    </exclusion>
</exclusions>

修改Spark配置，强制指定Hadoop版本
在SparkConf中添加版本号配置，跳过ShimLoader的版本检测：
```
val conf = new SparkConf()
  // 其他配置...
  .set("spark.hadoop.hadoop.version", "3.0.0")
```
替换Hadoop安装包为官方版本
将当前使用的CDH定制版Hadoop替换为Apache官方发布的Hadoop 3.0.0 Windows版本，确保路径中无CDH相关标识。
清理本地仓库缓存
执行Maven命令清理缓存并重新拉取依赖：
```
mvn clean install -U
```

内容的提问来源于stack exchange，提问作者Peter Ivony

Windows10下初始化Spark实例报错：无法识别Hadoop主版本号

SparkSession初始化代码 #

错误日志 #

SparkSession初始化代码

错误日志