升级Scylla Driver后Spark任务报错，求兼容版本方案

背景 #

我们在Spark应用中使用Scylla Driver搭配ScyllaDB，同时集成Spark Cassandra Connector。原Scylla Driver版本为3.7.1-scylla-2（发布于2019年底），尝试升级至最新的3.11.2.0后，Spark任务运行时抛出方法找不到的错误。

原POM依赖 #

<dependency>
    <groupId>com.scylladb</groupId>
    <artifactId>scylla-driver-core</artifactId>
    <version>3.7.1-scylla-2</version>
</dependency>
<dependency>
    <groupId>com.scylladb</groupId>
    <artifactId>scylla-driver-mapping</artifactId>
    <version>3.7.1-scylla-2</version>
</dependency>
<dependency>
    <groupId>com.datastax.spark</groupId>
    <artifactId>spark-cassandra-connector-unshaded_2.12</artifactId>
    <version>2.4.3</version>
</dependency>

升级后的POM依赖 #

<dependency>
    <groupId>com.scylladb</groupId>
    <artifactId>scylla-driver-core</artifactId>
    <version>3.11.2.0</version>
</dependency>
<dependency>
    <groupId>com.scylladb</groupId>
    <artifactId>scylla-driver-mapping</artifactId>
    <version>3.11.2.0</version>
</dependency>
<dependency>
    <groupId>com.datastax.spark</groupId>
    <artifactId>spark-cassandra-connector-unshaded_2.12</artifactId>
    <version>2.4.3</version>
</dependency>

报错信息 #

2023-03-17 00:11:17,122 ERROR [org.apache.spark.executor.Executor] - <Exception in task 1.0 in stage 0.0 (TID 1)>
java.lang.NoSuchMethodError: com.datastax.driver.core.Metadata.getReplicas(Ljava/lang/String;Ljava/nio/ByteBuffer;)Ljava/util/Set;
at com.datastax.spark.connector.cql.LocalNodeFirstLoadBalancingPolicy.findReplicas(LocalNodeFirstLoadBalancingPolicy.scala:48)
at com.datastax.spark.connector.cql.LocalNodeFirstLoadBalancingPolicy.tokenAwareQueryPlan(LocalNodeFirstLoadBalancingPolicy.scala:57)
at com.datastax.spark.connector.cql.LocalNodeFirstLoadBalancingPolicy.newQueryPlan(LocalNodeFirstLoadBalancingPolicy.scala:73)
at com.datastax.driver.core.policies.PagingOptimizingLoadBalancingPolicy.newQueryPlan(PagingOptimizingLoadBalancingPolicy.java:69)
at com.datastax.driver.core.RequestHandler.<init>(RequestHandler.java:147)
at com.datastax.driver.core.SessionManager.execute(SessionManager.java:715)
at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:148)
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.datastax.spark.connector.cql.SessionProxy.invoke(SessionProxy.scala:37)
at com.sun.proxy.$Proxy13.execute(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.datastax.spark.connector.cql.SessionProxy.invoke(SessionProxy.scala:37)
at com.sun.proxy.$Proxy26.execute(Unknown Source)
at com.ebates.pcs.commons.sparkutil.ListingSparkUtils.getListingData(ListingSparkUtils.java:471)
at com.ebates.rcp.itemmaster.sparkjobs.MerchantPushJob$1.apply(MerchantPushJob.java:333)
at com.ebates.rcp.itemmaster.sparkjobs.MerchantPushJob$1.apply(MerchantPushJob.java:318)
at com.datastax.spark.connector.cql.CassandraConnector.$anonfun$withSessionDo$1(CassandraConnector.scala:112)
at com.datastax.spark.connector.cql.CassandraConnector.closeResourceAfterUse(CassandraConnector.scala:145)
at com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:111)
at com.ebates.rcp.itemmaster.sparkjobs.MerchantPushJob.lambda$merchantPush$d9b9cfa9$1(MerchantPushJob.java:318)
at org.apache.spark.api.java.JavaRDDLike.$anonfun$foreachPartition$1(JavaRDDLike.scala:219)
at org.apache.spark.api.java.JavaRDDLike.$anonfun$foreachPartition$1$adapted(JavaRDDLike.scala:219)
at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2(RDD.scala:1020)
at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2$adapted(RDD.scala:1020)
at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2236)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

问题分析 #

该错误是版本兼容性问题导致：Spark Cassandra Connector 2.4.3依赖的Datastax Java Driver版本与Scylla Driver 3.11.2.0不匹配。Connector 2.4.3调用的com.datastax.driver.core.Metadata.getReplicas(String, ByteBuffer)方法，在Scylla Driver 3.11.x版本中已被修改或移除，导致运行时无法找到该方法。

解决方案 #

根据当前Spark版本，提供两种兼容版本组合方案：

方案1：保留Spark 2.4.x（不升级Connector） #

继续使用Spark Cassandra Connector 2.4.3，搭配Scylla Driver 3.10.0-scylla-1（这是兼容Connector 2.4.x的最高Scylla Driver版本），修改后的POM依赖如下：

<dependency>
    <groupId>com.scylladb</groupId>
    <artifactId>scylla-driver-core</artifactId>
    <version>3.10.0-scylla-1</version>
</dependency>
<dependency>
    <groupId>com.scylladb</groupId>
    <artifactId>scylla-driver-mapping</artifactId>
    <version>3.10.0-scylla-1</version>
</dependency>
<dependency>
    <groupId>com.datastax.spark</groupId>
    <artifactId>spark-cassandra-connector-unshaded_2.12</artifactId>
    <version>2.4.3</version>
</dependency>

方案2：升级至Spark 3.x（同步升级Connector） #

如果允许升级Spark到3.x系列，可使用Spark Cassandra Connector 3.0.0+（例如3.2.0）搭配Scylla Driver 3.11.2.0，示例POM依赖：

<dependency>
    <groupId>com.scylladb</groupId>
    <artifactId>scylla-driver-core</artifactId>
    <version>3.11.2.0</version>
</dependency>
<dependency>
    <groupId>com.scylladb</groupId>
    <artifactId>scylla-driver-mapping</artifactId>
    <version>3.11.2.0</version>
</dependency>
<dependency>
    <groupId>com.datastax.spark</groupId>
    <artifactId>spark-cassandra-connector-unshaded_2.12</artifactId>
    <version>3.2.0</version>
</dependency>

验证建议 #

• 升级后执行测试任务，验证ScyllaDB连接、数据读写功能正常
• 使用mvn dependency:tree命令排查依赖树，确保无Driver版本冲突

内容的提问来源于stack exchange，提问作者Steven Wu