Spark-submit运行含Avatica JDBC驱动的Druid连接任务失败求助

API 探路者

2026-8-24

问题描述

我正在编写Spark任务，通过Avatica JDBC驱动连接Druid执行查询操作。本地运行Scala代码可正常连接Druid，但构建Fat Jar后用spark-submit运行，或通过spark-submit --jars指定驱动Jar时，均出现如下错误：

Exception in thread "main" org.apache.commons.dbcp.SQLNestedException: Cannot load JDBC driver class 'org.apache.calcite.avatica.remote.Driver'
        at org.apache.commons.dbcp.BasicDataSource.createConnectionFactory(BasicDataSource.java:1429)
        at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1371)
        at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
        at DbConnection.<init>(SimpleApp.scala:43)
        at SimpleApp$.main(SimpleApp.scala:26)
        at SimpleApp.main(SimpleApp.scala)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
        at java.base/java.lang.reflect.Method.invoke(Method.java:577)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.calcite.avatica.remote.Driver
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:383)
        at java.base/java.lang.Class.forName(Class.java:376)
        at org.apache.commons.dbcp.BasicDataSource.createConnectionFactory(BasicDataSource.java:1415)
        ... 15 more

已确认Fat Jar中包含org.apache.calcite.avatica:avatica-core:1.21.0依赖，刚接触Spark，求解决思路。

示例代码：

object SimpleApp {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder().appName("SimpleApp").getOrCreate()
    val a = 15
    val b = 30
    val result = a * b
    println(s"Result is $result")
    val url = "jdbc:avatica:remote:url=http://<druidBrokerUrl>/druid/v2/sql/avatica/"
    val sql = "SELECT * FROM table LIMIT 5"
    val connectionProperties = new Properties
    val connection = DriverManager.getConnection(url, connectionProperties)
    val statement = connection.createStatement
    val resultSet = statement.executeQuery(sql)
    while (resultSet.next) {
      println(resultSet.getString(5))
    }
    spark.stop()
}

解决思路

调整Spark类加载优先级
Spark默认优先加载自身自带的依赖，若集群中存在版本冲突的Calcite/Avatica依赖，会覆盖你打包的版本。可在spark-submit时添加参数强制优先加载用户类路径：

spark-submit --conf spark.driver.userClassPathFirst=true --conf spark.executor.userClassPathFirst=true ...

注意：该参数可能引发其他依赖冲突，需测试验证。

显式注册JDBC驱动
在代码中手动加载驱动类，避免DriverManager自动查找失败：

Class.forName("org.apache.calcite.avatica.remote.Driver")
val connection = DriverManager.getConnection(url, connectionProperties)

排查并解决依赖冲突

用jar -tf your-fat-jar.jar | grep avatica确认依赖文件完整

用构建工具的依赖分析命令（如gradle dependencies或mvn dependency:tree）排查是否存在多版本Calcite/Avatica依赖，排除Spark自带的冲突依赖：
Gradle示例：

dependencies {
    implementation("org.apache.calcite.avatica:avatica-core:1.21.0") {
        exclude group: "org.apache.calcite", module: "calcite-core"
    }
}

Maven示例：

<dependency>
    <groupId>org.apache.calcite.avatica</groupId>
    <artifactId>avatica-core</artifactId>
    <version>1.21.0</version>
    <exclusions>
        <exclusion>
            <groupId>org.apache.calcite</groupId>
            <artifactId>calcite-core</artifactId>
        </exclusion>
    </exclusions>
</dependency>

改用Spark原生JDBC API
Spark内置的JDBC读取API更适配Spark运行环境，可避免直接使用DriverManager带来的类加载问题：

val df = spark.read
    .format("jdbc")
    .option("url", url)
    .option("dbtable", "(SELECT * FROM table LIMIT 5) AS tmp")
    .option("driver", "org.apache.calcite.avatica.remote.Driver")
    .load()

df.show()

验证版本兼容性
检查Druid版本与Avatica版本是否匹配，Druid官方文档会指定对应的Avatica版本，版本不兼容可能导致驱动初始化失败。

内容的提问来源于stack exchange，提问作者Dyson