Hadoop集群运行Mahout遭遇java.lang.ClassNotFoundException问题

错误信息 #

Error: java.lang.ClassNotFoundException:
org.apache.mahout.cf.taste.model.DataModel
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2625)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2590)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2686)
at org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:211)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:614)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)

该代码在本地环境运行正常，以下是原pom.xml配置：

<dependencies>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>3.3.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-core</artifactId>
        <version>1.2.1</version>
    </dependency>
    <dependency>
        <groupId>org.projectlombok</groupId>
        <artifactId>lombok</artifactId>
        <version>1.18.24</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.mahout</groupId>
        <artifactId>mahout-core</artifactId>
        <version>0.9</version>
    </dependency>
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-math3</artifactId>
        <version>3.6.1</version>
    </dependency>
</dependencies>

问题分析与解决方案 #

核心问题 #

1. Hadoop依赖版本冲突：同时引入hadoop-client:3.3.1和hadoop-core:1.2.1，两个版本跨度极大，Hadoop 3.x已不再使用hadoop-core这个artifact，该依赖完全多余，会引发类加载混乱。
2. Mahout依赖未随应用打包：本地运行时依赖存在于本地仓库，集群运行时未将Mahout的类包打进应用jar，导致节点找不到DataModel类。
3. 版本兼容性：Mahout 0.9是较旧版本，与Hadoop 3.3.1存在兼容性风险，部分类可能无法正常加载。

修复步骤 #

• 移除冲突依赖：删除hadoop-core依赖，仅保留hadoop-client:3.3.1即可。
• 添加打包插件：使用maven-shade-plugin将所有依赖打包成一个包含所有类的可执行jar，确保集群节点能找到Mahout相关类。
• （可选）升级Mahout版本：若兼容性问题持续，可尝试升级到Mahout 14.0及以上版本（该版本开始支持Hadoop 3.x）。

修改后的pom.xml示例 #

<dependencies>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>3.3.1</version>
    </dependency>
    <dependency>
        <groupId>org.projectlombok</groupId>
        <artifactId>lombok</artifactId>
        <version>1.18.24</version>
        <scope>provided</scope>
    </dependency>
    <!-- 若使用兼容Hadoop3.x的版本，替换为以下依赖 -->
    <!-- <dependency>
        <groupId>org.apache.mahout</groupId>
        <artifactId>mahout-core</artifactId>
        <version>14.0</version>
    </dependency> -->
    <dependency>
        <groupId>org.apache.mahout</groupId>
        <artifactId>mahout-core</artifactId>
        <version>0.9</version>
    </dependency>
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-math3</artifactId>
        <version>3.6.1</version>
    </dependency>
</dependencies>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.2.4</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <transformers>
                            <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <!-- 替换为你的主类全路径 -->
                                <mainClass>com.yourpackage.YourMainClass</mainClass>
                            </transformer>
                        </transformers>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

执行mvn clean package后，使用生成的带依赖的jar包提交到Hadoop集群运行即可。

内容的提问来源于stack exchange，提问作者Ankur Chauhan