PySpark应用无法加载Cassandra数据：Guava Jar版本冲突求助
2026-8-20
PySpark连接Cassandra时Guava版本冲突导致NoSuchMethodError问题
问题现象
PySpark应用无法从Cassandra加载数据，已定位到Guava Jar包版本冲突问题，尝试过多个版本的Guava（包括DataStax的shaded版本），仍无法找到适配的正确版本，运行时抛出java.lang.NoSuchMethodError错误。
环境信息
- Scala版本：2.11.12
- Spark版本：2.3.2.3.1.4.41-3
- 提交任务时使用的Jar包：
- spark-cassandra-connector_2.11-2.3.2.jar
- cassandra-driver-core-3.0.0.jar
- commons-configuration-1.7.jar
- 尝试过的Guava版本：java-driver-shaded-guava-25.1-jre.jar、Guava 19/24/31
错误日志
Error: File "cass.py", line 6, in <module> data_df = ss.read.format("org.apache.spark.sql.cassandra").options(keyspace="xxxxx",table="xxxxx").load() File "/disk-3/hadoop/yarn/local/usercache/vdfidt1/appcache/application_1660663467107_0171/container_e68_1660663467107_0171_01_000001/pyspark.zip/pyspark/sql/readwriter.py", line 172, in load File "/disk-3/hadoop/yarn/local/usercache/vdfidt1/appcache/application_1660663467107_0171/container_e68_1660663467107_0171_01_000001/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__ File "/disk-3/hadoop/yarn/local/usercache/vdfidt1/appcache/application_1660663467107_0171/container_e68_1660663467107_0171_01_000001/pyspark.zip/pyspark/sql/utils.py", line 63, in deco File "/disk-3/hadoop/yarn/local/usercache/vdfidt1/appcache/application_1660663467107_0171/container_e68_1660663467107_0171_01_000001/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o103.load. py4j.protocol.Py4JJavaError: An error occurred while calling o103.load. : java.lang.NoSuchMethodError: com.google.common.base.Objects.firstNonNull(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; at com.datastax.driver.core.policies.Policies$Builder.build(Policies.java:285) at com.datastax.driver.core.Cluster$Builder.getConfiguration(Cluster.java:1246) at com.datastax.driver.core.Cluster.<init>(Cluster.java:116) at com.datastax.driver.core.Cluster.buildFrom(Cluster.java:181) at com.datastax.driver.core.Cluster$Builder.build(Cluster.java:1264) at com.datastax.spark.connector.cql.DefaultConnectionFactory$.createCluster(CassandraConnectionFactory.scala:131) at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:159) at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$8.apply(CassandraConnector.scala:154) at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$8.apply(CassandraConnector.scala:154) at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:32) at com.datastax.spark.connector.cql.RefCountedCache.syncAcquire(RefCountedCache.scala:69) at com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:57) at com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:79) at com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:111) at com.datastax.spark.connector.rdd.partitioner.dht.TokenFactory$.forSystemLocalPartitioner(TokenFactory.scala:98) at org.apache.spark.sql.cassandra.CassandraSourceRelation$.apply(CassandraSourceRelation.scala:272) at org.apache.spark.sql.cassandra.DefaultSource.createRelation(DefaultSource.scala:56) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:341) at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:239) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:227) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:164) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:750)
相关代码
PySpark脚本（cass.py）
import pyspark from pyspark.sql import SparkSession,SQLContext ss = SparkSession.builder.appName("pyspark cassandra").getOrCreate() data_df = ss.read.format("org.apache.spark.sql.cassandra")\ .options(keyspace="reg_idt1_vdf",table="veh")\ .load() data_df.show() ss.stop()
Spark提交脚本（cass.sh）
function setJars() { i=0 for file in /home/tst1/*.jar do JARS=${JARS},${file} done } setJars spark-submit \ --name 'pyspark cassandra connector' \ --master yarn \ --deploy-mode cluster \ --driver-memory 2g \ --executor-memory 2g \ --num-executors 20 \ --jars ${JARS} \ --conf "spark.yarn.maxAppAttempts=1" \ --conf "spark.cassandra.connection.host=xx.xx.xx.xx" \ --conf "spark.cassandra.auth.username=xxxxx" \ --conf "spark.cassandra.auth.password=xxxxxx" \ --conf "spark.dynamicAllocation.enabled=true" \ --conf "spark.dynamicAllocation.maxExecutors=100" \ --conf "spark.dynamicAllocation.minExecutors=10" \ --conf "spark.executor.cores=2" \ --conf "spark.dynamicAllocation.executorIdleTimeout=500s" \ --conf "spark.authenticate=true" \ /home/tst1/cass.py exit 0
注意：提交脚本中原
$/home/tst1/cass.py多了一个$，已修正为/home/tst1/cass.py
问题原因与解决方案
原因分析
错误中的com.google.common.base.Objects.firstNonNull方法在Guava 19及以上版本中被迁移到了com.google.common.base.MoreObjects类中，而你使用的cassandra-driver-core-3.0.0.jar是基于旧版Guava（16.0.1）开发的，当应用加载了高版本Guava时，就会出现方法找不到的错误。
解决方案
使用匹配的Guava版本
移除所有手动添加的Guava Jar（包括shaded版本），替换为Guava 16.0.1.jar——这是cassandra-driver-core-3.0.0官方依赖的版本，能完美兼容当前的Spark Cassandra Connector和Driver版本。优先加载用户Jar包
在Spark提交脚本中添加以下配置，确保用户提供的Jar包优先于Spark自带的依赖加载，避免版本覆盖：--conf "spark.driver.userClassPathFirst=true" \ --conf "spark.executor.userClassPathFirst=true" \检查Jar包依赖冗余
确保/home/tst1/目录下没有多个版本的Guava Jar，避免提交时同时加载冲突版本。
内容的提问来源于stack exchange，提问作者user19874233
相关产品推荐
相关产品推荐
