在Java中运行Apache Spark Word2Vec时遭遇NoClassDefFoundError求助

问题重现 #

代码片段 #

SparkSession spark = SparkSession.builder().appName("Synonym Recommender")
                    .config("spark.master", "local")
                    .getOrCreate();

JavaRDD<String> lines = spark.read().textFile(Paths.get("src/main/resources/static/text8.txt").toString()).toJavaRDD();
JavaRDD<Iterable<String>> wordsIterable = lines.map(new Function<String, Iterable<String>>() {
    public Iterable<String> call(String s) throws Exception {
        String[] words = s.split(" ");
        Iterable<String> output = Arrays.asList(words);
        return output;
    }
});
Word2Vec vec = new Word2Vec();
vecModel = vec.fit(wordsIterable);

核心报错信息 #

java.lang.NoClassDefFoundError: org/codehaus/janino/InternalCompilerException

完整堆栈跟踪：

Caused by: java.lang.NoClassDefFoundError: org/codehaus/janino/InternalCompilerException
at org.apache.spark.sql.catalyst.expressions.objects.GetExternalRowField.<init>(objects.scala:1850) ~[spark-catalyst_2.13-3.3.1.jar:3.3.1]
at org.apache.spark.sql.catalyst.encoders.RowEncoder$.$anonfun$serializerFor$3(RowEncoder.scala:195) ~[spark-catalyst_2.13-3.3.1.jar:3.3.1]
at scala.collection.ArrayOps$.flatMap$extension(ArrayOps.scala:986) ~[scala-library-2.13.0.jar:na]
at org.apache.spark.sql.catalyst.encoders.RowEncoder$.serializerFor(RowEncoder.scala:192) ~[spark-catalyst_2.13-3.3.1.jar:3.3.1]
at org.apache.spark.sql.catalyst.encoders.RowEncoder$.apply(RowEncoder.scala:73) ~[spark-catalyst_2.13-3.3.1.jar:3.3.1]
at org.apache.spark.sql.catalyst.encoders.RowEncoder$.apply(RowEncoder.scala:81) ~[spark-catalyst_2.13-3.3.1.jar:3.3.1]
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$1(Dataset.scala:92) ~[spark-sql_2.13-3.3.1.jar:3.3.1]
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779) ~[spark-sql_2.13-3.3.1.jar:3.3.1]
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:89) ~[spark-sql_2.13-3.3.1.jar:3.3.1]
at org.apache.spark.sql.SparkSession.baseRelationToDataFrame(SparkSession.scala:444) ~[spark-sql_2.13-3.3.1.jar:3.3.1]
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:228) ~[spark-sql_2.13-3.3.1.jar:3.3.1]
at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:210) ~[spark-sql_2.13-3.3.1.jar:3.3.1]
at scala.Option.getOrElse(Option.scala:202) ~[scala-library-2.13.0.jar:na]
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:210) ~[spark-sql_2.13-3.3.1.jar:3.3.1]
at org.apache.spark.sql.DataFrameReader.text(DataFrameReader.scala:645) ~[spark-sql_2.13-3.3.1.jar:3.3.1]
at org.apache.spark.sql.DataFrameReader.textFile(DataFrameReader.scala:682) ~[spark-sql_2.13-3.3.1.jar:3.3.1]
at org.apache.spark.sql.DataFrameReader.textFile(DataFrameReader.scala:654) ~[spark-sql_2.13-3.3.1.jar:3.3.1]
at org.apache.spark.sql.DataFrameReader$textFile.call(Unknown Source) ~[na:na]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47) ~[groovy-2.5.14.jar:2.5.14]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115) ~[groovy-2.5.14.jar:2.5.14]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127) ~[groovy-2.5.14.jar:2.5.14]
at com.tcwb.classification.services.USMLService.loadWord2VecModel(testapp.groovy:591) ~[classes/:na]
at com.tcwb.classification.services.USMLService.postConstruct(testapp.groovy:76) ~[classes/:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389) ~[spring-beans-5.3.6.jar:5.3.6]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333) ~[spring-beans-5.3.6.jar:5.3.6]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157) ~[spring-beans-5.3.6.jar:5.3.6]
... 56 common frames omitted

当前依赖配置 #

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-mllib_2.13</artifactId>
    <version>3.3.1</version>
    <exclusions>
        <exclusion>
            <artifactId>janino</artifactId>
            <groupId>org.codehaus.janino</groupId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>org.codehaus.janino</groupId>
    <artifactId>janino</artifactId>
    <version>3.1.9</version>
</dependency>

错误原因 #

该报错由Janino依赖版本不兼容导致：
Spark 3.3.1原生依赖Janino 3.0.21版本，你排除了Spark自带的依赖后，手动引入的3.1.9版本与Spark内部类结构不匹配，导致运行时找不到指定类。

解决方案 #

方案1：移除Janino排除，使用Spark自带依赖 #

修改pom.xml，删除spark-mllib中的Janino排除配置，让Maven自动引入Spark兼容的版本：

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-mllib_2.13</artifactId>
    <version>3.3.1</version>
    <!-- 移除Janino排除配置 -->
</dependency>

方案2：手动引入匹配版本的Janino #

如果必须单独管理Janino依赖，确保引入与Spark 3.3.1兼容的3.0.21版本：

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-mllib_2.13</artifactId>
    <version>3.3.1</version>
    <exclusions>
        <exclusion>
            <artifactId>janino</artifactId>
            <groupId>org.codehaus.janino</groupId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>org.codehaus.janino</groupId>
    <artifactId>janino</artifactId>
    <version>3.0.21</version>
</dependency>

轻量/预训练同义词生成替代方案 #

如果Spark Word2Vec过于繁重，推荐以下Java方案：

• Apache Commons Text：基于WordNet实现同义词查询，无需训练，轻量易用，核心类为WordNetSynonymProvider。
• DeepLearning4j：支持加载预训练的Word2Vec/GloVe/FastText模型，无需自行训练，适合快速集成。
• Stanford CoreNLP：提供同义词扩展功能，可结合其词法分析模块实现查询。
• FastText Java实现：Facebook开源的FastText有Java版本，可加载预训练模型快速生成同义词。

内容的提问来源于stack exchange，提问作者SWS3D