Scala 2.11集成Embedded-Kafka测试时遇NoClassDefFoundError求助

在Scala 2.11项目的测试环节集成Embedded-Kafka时，启动Kafka服务抛出以下错误：

java.lang.NoClassDefFoundError: org/apache/zookeeper/AsyncCallback$MultiCallback

所有Kafka相关依赖版本已保持一致，以下是相关配置和报错详情：

依赖配置（Maven） #

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql-kafka-0-10_2.11</artifactId>
    <version>2.4.7</version>
</dependency>
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>2.4.0</version>
</dependency>
<dependency>
    <groupId>io.github.embeddedkafka</groupId>
    <artifactId>embedded-kafka_2.11</artifactId>
    <version>2.4.0</version>
    <scope>test</scope>
</dependency>

Embedded-Kafka启停代码 #

implicit val embedKafkaConfig = EmbeddedKafkaConfig(kafkaPort = 7000, zooKeeperPort = 7001)
implicit val stringDeserializer = new StringDeserializer()
implicit val stringSerializer = new StringSerializer

Before("") { _: Scenario =>
  EmbeddedKafka.start()(embedKafkaConfig)
}
After("") { _: Scenario =>
  EmbeddedKafka.stop()
}

完整报错堆栈 #

kafka.server.KafkaServer - Fatal error during KafkaServer startup. Prepare to shutdown
java.lang.NoClassDefFoundError: org/apache/zookeeper/AsyncCallback$MultiCallback
    at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1857)
    at kafka.server.KafkaServer.kafka$server$KafkaServer$$createZkClient$1(KafkaServer.scala:374)
    at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:399)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:207)
    at net.manub.embeddedkafka.ops.KafkaOps$class.startKafka(kafkaOps.scala:52)
    at net.manub.embeddedkafka.EmbeddedKafka$.startKafka(EmbeddedKafka.scala:50)
    at net.manub.embeddedkafka.ops.KafkaOps$class.startKafka(kafkaOps.scala:60)
    at net.manub.embeddedkafka.EmbeddedKafka$.startKafka(EmbeddedKafka.scala:50)
    at net.manub.embeddedkafka.ops.RunningKafkaOps$class.startKafka(kafkaOps.scala:88)
    at net.manub.embeddedkafka.EmbeddedKafka$.startKafka(EmbeddedKafka.scala:50)
    at net.manub.embeddedkafka.EmbeddedKafka$.start(EmbeddedKafka.scala:68)

问题原因 #

该错误由ZooKeeper依赖版本冲突导致：

• Spark SQL Kafka 2.4.7依赖的ZooKeeper版本为3.4.x系列，而Kafka 2.4.0需要的ZooKeeper版本是3.5.x，AsyncCallback$MultiCallback类正是在3.5.x版本中新增的。
• 项目构建时低版本ZooKeeper jar被优先加载，覆盖了Kafka所需的高版本，导致启动时找不到目标类。

解决方案 #

方案1：显式指定兼容的ZooKeeper版本 #

在Maven依赖中添加ZooKeeper依赖，选择与Kafka 2.4.0兼容的3.5.x版本（如3.5.7），并设置为测试环境生效：

<dependency>
    <groupId>org.apache.zookeeper</groupId>
    <artifactId>zookeeper</artifactId>
    <version>3.5.7</version>
    <scope>test</scope>
    <exclusions>
        <exclusion>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
        </exclusion>
    </exclusions>
</dependency>

注：排除slf4j-log4j12是为了避免日志依赖冲突。

方案2：排除Spark依赖中的ZooKeeper #

如果测试环境不需要Spark自带的ZooKeeper，可以在spark-sql-kafka-0-10_2.11依赖中排除ZooKeeper，让Embedded-Kafka的依赖版本生效：

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql-kafka-0-10_2.11</artifactId>
    <version>2.4.7</version>
    <exclusions>
        <exclusion>
            <groupId>org.apache.zookeeper</groupId>
            <artifactId>zookeeper</artifactId>
        </exclusion>
    </exclusions>
</dependency>

验证方法 #

执行mvn dependency:tree命令，查看ZooKeeper的最终依赖版本，确保是3.5.x系列且无低版本依赖被引入。

内容的提问来源于stack exchange，提问作者gabby