使用Sqoop Eval读取MySQL表时出现NoClassDefFoundError错误求助

执行命令与错误输出 #

执行的Sqoop命令 #

h-user@h-primary:~/sqoop-1.4.7/bin$ sqoop eval --connect jdbc:mysql://localhost/HADOOP_BANK_DATA_POC --username root --password xxxxxxxx --query "select * from E_INFO"

终端错误输出 #

Warning: /home/h-user/sqoop-1.4.7/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /home/h-user/sqoop-1.4.7/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/h-user/sqoop-1.4.7/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/h-user/sqoop-1.4.7/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
/usr/local/hadoop/libexec/hadoop-functions.sh: line 2366: HADOOP_ORG.APACHE.SQOOP.SQOOP_USER: invalid variable name
/usr/local/hadoop/libexec/hadoop-functions.sh: line 2461: HADOOP_ORG.APACHE.SQOOP.SQOOP_OPTS: invalid variable name
2022-12-23 17:13:06,493 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
2022-12-23 17:13:06,683 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/avro/LogicalType
    at org.apache.sqoop.manager.DefaultManagerFactory.accept(DefaultManagerFactory.java:67)
    at org.apache.sqoop.ConnFactory.getManager(ConnFactory.java:184)
    at org.apache.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:272)
    at org.apache.sqoop.tool.EvalSqlTool.run(EvalSqlTool.java:56)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: java.lang.ClassNotFoundException: org.apache.avro.LogicalType
    at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    ... 10 more
h-user@h-primary:~/s

错误原因及解决方法 #

1. HBase/HCatalog/Accumulo/ZooKeeper路径警告 #

这些是非致命提示，仅因Sqoop默认检查大数据组件的安装路径，若不使用相关功能可直接忽略。如需消除警告：

• 在环境变量中设置对应组件的安装路径，例如：export HBASE_HOME=/your/hbase/path，或在Sqoop配置文件conf/sqoop-env.sh中配置这些变量。

2. Hadoop变量名无效错误 #

Linux环境变量名不允许包含.，HADOOP_ORG.APACHE.SQOOP.SQOOP_USER和HADOOP_ORG.APACHE.SQOOP.SQOOP_OPTS属于无效变量名，是Sqoop与Hadoop版本不兼容导致的，解决方式：

• 修改Sqoop启动脚本bin/sqoop，调整SQOOP_USER和SQOOP_OPTS的变量名格式，避免包含.；
• 升级Sqoop至与当前Hadoop版本兼容的版本；
• 执行命令前手动设置有效变量：export SQOOP_USER=h-user、export SQOOP_OPTS=""。

3. NoClassDefFoundError: org/apache/avro/LogicalType #

这是致命错误，核心原因是Sqoop缺少Avro相关依赖包，解决步骤：

• 检查Sqoop的lib目录，确认是否存在avro-xxx.jar，若不存在，下载与Sqoop 1.4.7兼容的Avro Jar包（推荐avro-1.8.2及以上版本）放入该目录；
• 检查Hadoop的lib目录是否有冲突的Avro Jar包，若有替换为与Sqoop兼容的版本，避免版本冲突；
• 临时解决可通过设置SQOOP_CLASSPATH指定Avro Jar路径：export SQOOP_CLASSPATH=/path/to/avro.jar，再执行Sqoop命令。

内容的提问来源于stack exchange，提问作者Salva