It’s frustrating when a working setup breaks after version upgrades—let’s walk through the most common fixes for your scenario:

1. Double-Check Compatibility Fine Print #

While the connector’s major version aligns with Spark’s, sometimes minor versions have hidden gotchas. Spark 3.0.1 might have subtle differences that Connector 3.0 doesn’t fully account for. Verify that the connector’s release notes explicitly list Spark 3.0.1 as a supported version, or consider trying the next patch version of the connector (like 3.0.1) if it exists.

2. Hunt for Dependency Conflicts #

This is the #1 culprit after version jumps. Spark 3.0.1 and Connector 3.0 might pull in different versions of the underlying DataStax Cassandra driver, or leftover old connector JARs might be cluttering your classpath.

• For Maven, ensure your pom.xml explicitly declares the connector with the correct Scala version (_2.12) and excludes conflicting transitive dependencies:

  <dependency>
    <groupId>com.datastax.spark</groupId>
    <artifactId>spark-cassandra-connector_2.12</artifactId>
    <version>3.0.0</version>
    <exclusions>
      <exclusion>
        <groupId>com.datastax.cassandra</groupId>
        <artifactId>cassandra-driver-core</artifactId>
      </exclusion>
    </exclusions>
  </dependency>
  

• For SBT, add a dependency override to force a compatible driver version that works with both Spark 3.0.1 and AWS Keyspaces.

3. Verify AWS Keyspaces-Specific Configs #

Spark 3.0/Connector 3.0 might have changed how certain configs are parsed. Double-check these critical settings:

• Ensure SSL is enabled: spark.cassandra.connection.ssl.enabled=true
• Confirm the correct auth provider for IAM: spark.cassandra.auth.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain
• Verify the connection endpoint points to AWS Keyspaces’ regional URL (e.g., cassandra.us-east-1.amazonaws.com:9142)
• Check that spark.cassandra.connection.port is set to 9142 (the standard SSL port for Keyspaces)

4. Adapt Your Write Code to Spark 3.0 API Changes #

Some DataFrame write methods were deprecated or modified in Spark 3.0. Make sure your code follows the latest pattern:

import org.apache.spark.sql.SaveMode

df.write
  .format("org.apache.spark.sql.cassandra")
  .option("keyspace", "your_target_keyspace")
  .option("table", "your_target_table")
  .mode(SaveMode.Append) // Or Overwrite/Ignore as needed
  .save()

Avoid any deprecated syntax like using saveAsTable with Cassandra-specific options without the proper format.

5. Dig Into the Error Logs #

Don’t ignore the stack trace! Look for specific exceptions like:

• AuthenticationException: Indicates IAM credentials aren’t being picked up correctly (check your AWS credentials setup, or try explicitly setting them via spark.cassandra.auth.username/password for temporary testing)
• SchemaMismatchException: Your DataFrame’s schema doesn’t match the Keyspaces table (e.g., a Long in Spark vs int in Cassandra, or missing columns)
• ConnectionTimeoutException: Network issues or incorrect endpoint/port settings

6. Test a Gradual Rollback #

If all else fails, temporarily roll back to the working Spark 2.4.6 + Connector 2.5.2 setup to confirm the issue is indeed version-related. Then, incrementally upgrade one component at a time (first Spark to 3.0.1 with old connector, then connector to 3.0) to isolate which change caused the failure.

内容的提问来源于stack exchange，提问作者Alan Miranda