Spark 2.4.0环境下Amazon Deequ兼容依赖版本及正确pom文件求助

问题根因 #

你遇到的java.lang.NoClassDefFoundError: scala/runtime/java8/JFunction1$mcZD$sp报错由两类冲突共同导致：

1. 项目依赖中混入了多个版本的Scala类库，类定义不匹配
2. 手动引入的Jackson 2.11.0版本远高于Spark 2.4内置的Jackson版本，引发序列化类冲突

统一兼容版本清单 #

• Scala：2.11.12（所有后缀为_2.11的Scala生态依赖必须对齐此版本）
• Spark：2.4.0（和你现有运行环境保持一致）
• Deequ：0.1.3-spark-2.4（官方适配Spark 2.4 + Scala 2.11的稳定版，无版本冲突）
• Jackson：2.6.7.1（Spark 2.4 内置绑定版本，无需手动升级避免冲突）

修正后完整POM.xml #

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <artifactId>dq-test</artifactId>
  <groupId>com.package.name</groupId>
  <version>${revision}</version>
  <properties>
    <revision>0.1</revision>
    <scala.version>2.11.12</scala.version>
    <spark.version>2.4.0</spark.version>
    <deequ.version>0.1.3-spark-2.4</deequ.version>
    <jackson.version>2.6.7.1</jackson.version>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
  </properties>
  <dependencies>
    <!-- Spark核心依赖，scope设为provided避免打包时混入集群已有的类 -->
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.11</artifactId>
      <version>${spark.version}</version>
      <scope>provided</scope>
      <exclusions>
        <exclusion>
          <groupId>org.scala-lang</groupId>
          <artifactId>*</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.fasterxml.jackson.core</groupId>
          <artifactId>*</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.11</artifactId>
      <version>${spark.version}</version>
      <scope>provided</scope>
      <exclusions>
        <exclusion>
          <groupId>org.scala-lang</groupId>
          <artifactId>*</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.fasterxml.jackson.core</groupId>
          <artifactId>*</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <!-- 统一锁定Jackson版本 -->
    <dependency>
      <groupId>com.fasterxml.jackson.core</groupId>
      <artifactId>jackson-databind</artifactId>
      <version>${jackson.version}</version>
    </dependency>
    <dependency>
      <groupId>com.fasterxml.jackson.core</groupId>
      <artifactId>jackson-core</artifactId>
      <version>${jackson.version}</version>
    </dependency>
    <dependency>
      <groupId>com.fasterxml.jackson.core</groupId>
      <artifactId>jackson-annotations</artifactId>
      <version>${jackson.version}</version>
    </dependency>
    <!-- Deequ适配版本 -->
    <dependency>
      <groupId>com.amazon.deequ</groupId>
      <artifactId>deequ_2.11</artifactId>
      <version>${deequ.version}</version>
      <exclusions>
        <exclusion>
          <groupId>org.scala-lang</groupId>
          <artifactId>*</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.fasterxml.jackson.core</groupId>
          <artifactId>*</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>com.typesafe</groupId>
      <artifactId>config</artifactId>
      <version>1.2.1</version>
    </dependency>
    <dependency>
      <groupId>com.sun.mail</groupId>
      <artifactId>javax.mail</artifactId>
      <version>1.6.2</version>
    </dependency>
    <dependency>
      <groupId>com.sun.activation</groupId>
      <artifactId>javax.activation</artifactId>
      <version>1.2.0</version>
    </dependency>
    <!-- 显式引入统一版本的Scala依赖 -->
    <dependency>
      <groupId>org.scala-lang</groupId>
      <artifactId>scala-library</artifactId>
      <version>${scala.version}</version>
    </dependency>
  </dependencies>
  <build>
    <plugins>
      <plugin>
        <groupId>net.alchim31.maven</groupId>
        <artifactId>scala-maven-plugin</artifactId>
        <version>3.2.2</version>
        <configuration>
          <scalaVersion>${scala.version}</scalaVersion>
        </configuration>
        <executions>
          <execution>
            <id>scala-compile-first</id>
            <phase>process-resources</phase>
            <goals>
              <goal>compile</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.3</version>
        <configuration>
          <skipMain>true</skipMain> <!-- skip compile -->
          <skip>true</skip> <!-- skip testCompile -->
          <source>1.8</source>
          <target>1.8</target>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.7</version>
        <configuration>
          <skipTests>true</skipTests>
        </configuration>
      </plugin>
      <plugin>
        <artifactId>maven-assembly-plugin</artifactId>
        <configuration>
          <archive>
            <manifest>
              <mainClass>className</mainClass>
            </manifest>
          </archive>
          <descriptorRefs>
            <descriptorRef>jar-with-dependencies</descriptorRef>
          </descriptorRefs>
        </configuration>
        <executions>
          <execution>
            <id>make-assembly</id>
            <phase>package</phase>
            <goals>
              <goal>single</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

验证方法 #

执行命令mvn dependency:tree | grep scala，检查所有Scala相关依赖版本均为2.11.12，无其他版本混入后执行mvn clean package打包即可正常运行。

内容的提问来源于stack exchange，提问作者Anu Shivangi