AWS EMR中Sqoop连接MySQL报错：无法加载JDBC驱动类

问题描述 #

我是一名学生，正在使用AWS EMR，通过Sqoop连接MySQL时遇到错误，提示Could not load db driver class: com.mysql.jdbc.Driver，不清楚添加驱动的操作步骤，需求是连接成功后运行Pig命令，找不到AWS EMR添加驱动的分步教程。

执行的Sqoop命令及错误日志：

[hadoop@ip-172-31-88-249 ~]$ sqoop import --connect jdbc:mysql://retail-db.cjmn8pjxazxq.us-east-1.rds.amazonaws.com/retail --table customers --username admin --password ***** --target-dir SQP/input/sqoop1 -m 1
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
22/10/26 02:30:38 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
22/10/26 02:30:39 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
22/10/26 02:30:39 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
22/10/26 02:30:39 INFO tool.CodeGenTool: Beginning code generation
22/10/26 02:30:39 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver
java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver
    at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:875)
    at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:59)
    at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:763)
    at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:786)
    at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:289)
    at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:260)
    at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:246)
    at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:327)
    at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1872)
    at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1671)
    at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:501)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:252)

解决方案 #

一、安装MySQL JDBC驱动到EMR集群 #

方法1：集群启动时通过Bootstrap动作添加（推荐，永久生效）

• 下载对应版本的MySQL JDBC驱动包（如mysql-connector-java-8.0.30.jar，需与你的MySQL版本匹配），上传到AWS S3存储桶
• 创建Bootstrap脚本install-mysql-driver.sh，内容如下：

#!/bin/bash
# 复制驱动到Sqoop和Pig的lib目录
aws s3 cp s3://你的S3存储桶路径/mysql-connector-java-8.0.30.jar /usr/lib/sqoop/lib/
aws s3 cp s3://你的S3存储桶路径/mysql-connector-java-8.0.30.jar /usr/lib/pig/lib/
# 设置文件权限
chmod 644 /usr/lib/sqoop/lib/mysql-connector-java-8.0.30.jar /usr/lib/pig/lib/mysql-connector-java-8.0.30.jar

• 将脚本上传到S3，在创建EMR集群时，添加Bootstrap动作并指定该脚本的S3路径，集群启动后驱动会自动部署到对应目录

方法2：集群运行时手动安装（临时生效，重启集群后需重新操作）

1. 通过SSH登录到EMR主节点
2. 下载MySQL JDBC驱动：

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-8.0.30.jar

3. 复制驱动到Sqoop和Pig的lib目录：

sudo cp mysql-connector-java-8.0.30.jar /usr/lib/sqoop/lib/
sudo cp mysql-connector-java-8.0.30.jar /usr/lib/pig/lib/

4. 设置权限确保程序可读取：

sudo chmod 644 /usr/lib/sqoop/lib/mysql-connector-java-8.0.30.jar /usr/lib/pig/lib/mysql-connector-java-8.0.30.jar

二、验证驱动安装 #

重新执行Sqoop命令，建议使用-P参数交互式输入密码，避免明文泄露：

sqoop import --connect jdbc:mysql://retail-db.cjmn8pjxazxq.us-east-1.rds.amazonaws.com/retail --table customers --username admin -P --target-dir SQP/input/sqoop1 -m 1

三、Pig命令运行注意事项 #

• 如果Pig读取Sqoop导入到HDFS的数据，直接指定HDFS路径SQP/input/sqoop1即可
• 如果Pig需要直接连接MySQL，确保驱动已在/usr/lib/pig/lib/目录下，脚本中使用JDBC连接时，MySQL 8.0+版本推荐使用驱动类com.mysql.cj.jdbc.Driver替代旧的com.mysql.jdbc.Driver

内容的提问来源于stack exchange，提问作者Wing