nutch多版本执行inject命令报WebPage$Field类未找到错误求解决方案

问题说明 #

• 覆盖场景：二进制发布版、手动编译源码版、Git仓库拉取的源码版均出现相同问题
• 影响版本：Nutch 1.16、1.17、1.18
• 已排除项：使用默认配置，排除自定义配置错误导致的问题

编译阶段报错 #

执行ant runtime时出现Ivy相关类缺失警告，但最终构建成功，日志如下：

resolve-default:
[ivy:resolve] impossible to define new type: class not found: org.apache.ivy.plugins.resolver.SshResolver in [] nor Ivy classloader
[ivy:resolve] impossible to define new type: class not found: org.apache.ivy.plugins.signer.bouncycastle.OpenPGPSignatureGenerator in [] nor Ivy classloader
[ivy:resolve] impossible to define new type: class not found: org.apache.ivy.plugins.resolver.SFTPResolver in [] nor Ivy classloader
[ivy:resolve] impossible to define new type: class not found: org.apache.ivy.plugins.resolver.VfsResolver in [] nor Ivy classloader
[ivy:resolve] :: loading settings :: file = /home/user/Téléchargements/nutch/branch-1.18/ivy/ivysettings.xml
...
BUILD SUCCESSFUL
Total time: 36 seconds

运行阶段报错 #

执行bin/nutch inject base/crawldb urls/命令注入URL时任务失败，核心报错为org.apache.nutch.storage.WebPage$Field类找不到，日志如下：

2021-05-07 23:00:18,306 WARN  mapred.LocalJobRunner - job_local829319691_0001
java.lang.Exception: java.lang.NoClassDefFoundError: org/apache/nutch/storage/WebPage$Field
    at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
Caused by: java.lang.NoClassDefFoundError: org/apache/nutch/storage/WebPage$Field
    at org.apache.nutch.scoring.opic.OPICScoringFilter.<clinit>(OPICScoringFilter.java:59)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.nutch.plugin.Extension.getExtensionInstance(Extension.java:164)
    at org.apache.nutch.plugin.PluginRepository.getOrderedPlugins(PluginRepository.java:442)
    at org.apache.nutch.scoring.ScoringFilters.<init>(ScoringFilters.java:46)
    at org.apache.nutch.crawl.Injector$InjectMapper.setup(Injector.java:145)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
    at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.nutch.storage.WebPage$Field
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    at org.apache.nutch.plugin.PluginClassLoader.loadClassFromSystem(PluginClassLoader.java:104)
    at org.apache.nutch.plugin.PluginClassLoader.loadClassFromParent(PluginClassLoader.java:92)
    at org.apache.nutch.plugin.PluginClassLoader.loadClass(PluginClassLoader.java:72)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    ... 18 more
2021-05-07 23:00:19,027 INFO  mapreduce.Job - Job job_local829319691_0001 failed with state FAILED due to: NA
2021-05-07 23:00:19,490 INFO  mapreduce.Job - Counters: 17
    File System Counters
        FILE: Number of bytes read=3158671488622
        FILE: Number of bytes written=11186329660
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
    Map-Reduce Framework
        Map input records=0
        Map output records=0
        Map output bytes=0
        Map output materialized bytes=81536
        Input split bytes=982778
        Combine input records=0
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=1515713
        Total committed heap usage (bytes)=20589437779968
    File Input Format Counters 
        Bytes Read=0
2021-05-07 23:00:19,491 ERROR crawl.Injector - Injector job did not succeed, job status: FAILED, reason: NA
2021-05-07 23:00:19,498 ERROR crawl.Injector - Injector: java.lang.RuntimeException: Injector job did not succeed, job status: FAILED, reason: NA
    at org.apache.nutch.crawl.Injector.inject(Injector.java:444)
    at org.apache.nutch.crawl.Injector.run(Injector.java:571)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.nutch.crawl.Injector.main(Injector.java:535)

排查解决步骤 #

• 处理编译阶段Ivy警告
  1. 不要使用系统全局安装的高版本Ivy，优先使用Nutch源码自带的Ivy依赖，Nutch 1.x分支仅兼容Ivy 2.4.x版本
  2. 先执行ant clean清空旧构建产物，再执行ant ivy-clean清空Ivy本地缓存的依赖，之后重新执行ant runtime完成编译
• 解决运行时类找不到问题
  1. 检查conf/nutch-site.xml中的plugin.folders配置，确保路径指向当前编译生成的plugins目录，不要指向其他旧版本的插件目录
  2. 如果是本地模式运行，修改bin/nutch启动脚本，在CLASSPATH变量中额外加入build/classes目录和lib目录下的所有jar包
  3. 如果是集群模式运行，将build目录下的nutch-*.jar包上传到Hadoop的类路径下，或者提交任务时通过-libjars参数带入该jar包
• 基础环境排查
  1. 确认JDK版本为1.8，Nutch 1.16及以上版本不兼容JDK 1.7及以下、JDK 11及以上版本
  2. 不要在包含中文、特殊字符的路径下编译运行Nutch，可将源码移动到纯英文无特殊符号的路径下重新编译验证

内容的提问来源于stack exchange，提问作者Boulou