Spark-Cassandra连接器加载失败：ClassNotFoundException问题求助

环境配置：

• Spark 3.1.1
• Cassandra 4.0.11
• Scala 2.12.18
• Spark-Cassandra连接器版本：3.1.0（匹配Spark版本）

执行提交命令：

spark-submit dev_spark/titi.py \
  --packages com.datastax.spark:spark-cassandra-connector_2.12:3.1.0

出现错误：java.lang.ClassNotFoundException: Failed to find data source: org.apache.spark.sql.cassandra

完整错误栈：

Traceback (most recent call last): File "/home/ubuntu/dev_spark/titi.py", line 12, in <module> .load() 
File "/home/ubuntu/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 210, 
  in load File "/home/ubuntu/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1304, 
  in __call__ File "/home/ubuntu/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 111, 
  in deco File "/home/ubuntu/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 326, 
  in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o36.load. : 
  java.lang.ClassNotFoundException: Failed to find data source: org.apache.spark.sql.cassandra. 
  Please find packages at http://spark.apache.org/third-party-projects.html 
  at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:689) 
  at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSourceV2(DataSource.scala:743) 
  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:266) 
  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:226) 
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
  at java.lang.reflect.Method.invoke(Method.java:498) 
  at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) 
  at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) 
  at py4j.Gateway.invoke(Gateway.java:282) 
  at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) 
  at py4j.commands.CallCommand.execute(CallCommand.java:79) 
  at py4j.GatewayConnection.run(GatewayConnection.java:238) 
  at java.lang.Thread.run(Thread.java:750) 
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.cassandra.DefaultSource 
  at java.net.URLClassLoader.findClass(URLClassLoader.java:387) 
  at java.lang.ClassLoader.loadClass(ClassLoader.java:418) 
  at java.lang.ClassLoader.loadClass(ClassLoader.java:351) 
  at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$lookupDataSource$5(DataSource.scala:663) 
  at scala.util.Try$.apply(Try.scala:213) 
  at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$lookupDataSource$4(DataSource.scala:663) 
  at scala.util.Failure.orElse(Try.scala:224)

测试文件titi.py内容：

from pyspark.sql import SparkSession

spark = SparkSession.builder \
  .appName("CassandraIntegration") \
  .config("spark.cassandra.connection.host", "tp-hadoop-33") \
  .config("spark.cassandra.connection.port", "9042") \
  .getOrCreate()

df = spark.read \
  .format("org.apache.spark.sql.cassandra") \
  .options(table="request_2", keyspace="gdelt") \
  .load()

spark.stop()

注：主机名tp-hadoop-33、端口9042可用，Cassandra中gdelt keyspace和request_2表均存在。

1. 修正spark-submit命令参数顺序 #

--packages参数必须放在脚本文件名之前，否则Spark不会加载指定的依赖包。正确命令：

spark-submit \
  --packages com.datastax.spark:spark-cassandra-connector_2.12:3.1.0 \
  dev_spark/titi.py

2. 手动指定依赖包路径 #

如果网络问题导致自动下载依赖失败，可手动下载匹配Scala 2.12、Spark 3.1.x版本的连接器jar包，通过--jars参数指定本地路径：

spark-submit \
  --jars /path/to/spark-cassandra-connector_2.12-3.1.0.jar \
  dev_spark/titi.py

也可将jar包直接放入Spark安装目录下的jars文件夹，无需每次提交命令指定。

3. 简化数据源格式配置 #

Spark-Cassandra连接器支持简写的数据源格式cassandra，可修改代码中的格式配置：

df = spark.read \
  .format("cassandra") \
  .options(table="request_2", keyspace="gdelt") \
  .load()

4. 验证版本兼容性 #

确认连接器版本与Spark版本完全兼容：Spark 3.1.x对应连接器3.1.x系列，Scala版本必须严格匹配（此处2.12正确）。若问题仍存在，可尝试升级连接器到3.1.2版本（兼容Spark 3.1.1）。

内容的提问来源于stack exchange，提问作者laurentg13