PySpark写入Ignite数据库报错：org.apache.ignite.IgniteJdbcThinDriver类未找到

系统版本信息 #

python version: 3.9.13 
pyspark version: 3.2.1 
py4j version: 0.10.9.3 
java version:  OpenJDK Runtime Environment Zulu11.56+19-CA (build 11.0.15+10-LTS) OpenJDK 64-Bit Server VM Zulu11.56+19-CA (build 11.0.15+10-LTS, mixed mode) 
OS version: macOS Big Sur 11.5.1 Mac M1 mini 
Chip: Apple M1

环境变量 #

SPARK_HOME: /opt/homebrew/Cellar/apache-spark/3.3.0/libexec 
CLASSPATH: /Library/Java/Extensions/ignite-core-2.12.0.jar 
JAVA_HOME: /Applications/zulu11.56.19-ca-jdk11.0.15-macosx_aarch64/zulu-11.jdk/Contents/Home

代码实现 #

创建SparkSession #

spark = SparkSession.builder.config(
    "spark.jars", "/Library/Java/Extensions/ignite-core-2.12.0.jar"
).getOrCreate()

写入Ignite的代码 #

spark_data_frame.write.format("jdbc").option(
    "driver", "org.apache.ignite.IgniteJdbcThinDriver"
).option(
    "url", f"jdbc:ignite:thin://127.0.0.1:10800/"
).option(
    "dbtable", "user"
).mode(
    "append"
).save()

报错信息 #

virtual_env/lib/python3.9/site-packages/py4j/protocol.py, line326, in get_return_value
 raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o797.save.
: java.lang.ClassNotFoundException: org.apache.ignite.IgniteJdbcThinDriver
    at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
    at org.apache.spark.sql.execution.datasources.jdbc.DriverRegistry$.register(DriverRegistry.scala:46)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.$anonfun$driverClass$1(JDBCOptions.scala:101)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.$anonfun$driverClass$1$adapted(JDBCOptions.scala:101)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:101)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcOptionsInWrite.<init>(JDBCOptions.scala:218)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcOptionsInWrite.<init>(JDBCOptions.scala:222)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:46)
    at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:110)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:110)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:106)
    at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:481)
    at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:82)
    at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:481)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
    at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:457)
    at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:106)
    at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:93)
    at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:91)
    at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:128)
    at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:848)
    at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:382)
    at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:355)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:247)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
    at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
    at java.base/java.lang.Thread.run(Thread.java:829)

问题说明 #

本人对Spark不熟悉，尝试解决该问题多日未果。已在SparkSession中配置了包含org.apache.ignite.IgniteJdbcThinDriver的ignite-core-2.12.0.jar，但仍报类找不到错误。Ignite支持Java 8或11，PySpark也支持Java 11，我的Java是ARM版本的11，不确定是Java版本、ARM架构还是Spark配置问题，恳请帮忙解决。

解决方案 #

1. 确认驱动包完整性 #

先检查ignite-core-2.12.0.jar是否包含目标驱动类：

jar tf /Library/Java/Extensions/ignite-core-2.12.0.jar | grep IgniteJdbcThinDriver

如果无输出，说明该jar包不包含此驱动类，需下载包含该类的Ignite JDBC驱动包（如ignite-jdbc-2.12.0.jar）或完整Ignite发行包。

2. 调整Spark jar包配置方式 #

ARM架构Mac上仅通过spark.jars配置可能存在类加载问题，可尝试：

• 复制驱动包到Spark自带jars目录：

  cp /Library/Java/Extensions/ignite-core-2.12.0.jar $SPARK_HOME/jars/
  

  之后重启SparkSession，无需再配置spark.jars参数。
• 配置额外类路径：
  在SparkSession中同时指定驱动和执行器的类路径：

  spark = SparkSession.builder.config(
      "spark.driver.extraClassPath", "/Library/Java/Extensions/ignite-core-2.12.0.jar"
  ).config(
      "spark.executor.extraClassPath", "/Library/Java/Extensions/ignite-core-2.12.0.jar"
  ).getOrCreate()
  

• 提交任务时指定jar包：
  若用spark-submit，添加参数：

  spark-submit --jars /Library/Java/Extensions/ignite-core-2.12.0.jar your_script.py
  

3. 验证ARM架构兼容性 #

确认下载的Ignite jar包支持ARM架构。多数Java jar包跨架构通用，若怀疑兼容性，可重新下载Ignite官方针对Java 11的版本。

4. 检查环境变量有效性 #

确认CLASSPATH被Spark正确读取：

import os
print(os.environ.get('CLASSPATH'))

若未读取到，可在启动Python前显式设置：

export CLASSPATH="/Library/Java/Extensions/ignite-core-2.12.0.jar:$CLASSPATH"

内容的提问来源于stack exchange，提问作者htlbydgod