Scala IDE中Spark-SQL查询DSE Cassandra表报错排查

Let's break down your two main issues and walk through how to fix them step by step:

1. Why killr_video.videos isn't found in local IDE mode #

When you use dse spark-submit, DSE automatically handles all the heavy lifting: injecting the correct Spark-Cassandra connector dependencies, configuring DSE-specific Spark extensions, and setting up connection parameters to your Cassandra cluster. But in your IDE's local Spark session, none of this happens by default.

Here's what you need to do:

• Add the right DSE Spark dependencies to your IDE project (Maven/Gradle). Make sure the version matches your DSE cluster's version exactly. For example, if you're on DSE 6.8.x with Scala 2.12, add this to your pom.xml:

  <dependency>
      <groupId>com.datastax.dse</groupId>
      <artifactId>dse-spark-driver_2.12</artifactId>
      <version>6.8.22</version> <!-- Match your DSE version -->
  </dependency>
  

• Configure your SparkSession explicitly to connect to DSE Cassandra and enable DSE extensions:

  import org.apache.spark.sql.SparkSession
  
  val spark = SparkSession.builder()
    .appName("DSE Cassandra Test")
    .master("local[*]") // Use local[*] to utilize all CPU cores, avoids resource bottlenecks
    .config("spark.cassandra.connection.host", "127.0.0.1") // Your DSE node IP
    .config("spark.sql.extensions", "com.datastax.spark.connector.CassandraSparkExtensions")
    // Add auth configs if your DSE cluster uses authentication:
    // .config("spark.cassandra.auth.username", "your-dse-username")
    // .config("spark.cassandra.auth.password", "your-dse-password")
    .getOrCreate()
  

• Verify version compatibility: Ensure your IDE's Spark version is identical to the Spark version bundled with your DSE cluster. Mismatched versions will cause all sorts of unexpected errors, including missing tables.

2. Fixing MapOutputTracker InterruptedException with spark://127.0.0.1:7077 #

This error usually points to a mismatch between your IDE's Spark setup and the DSE Spark cluster, or network/resource issues. Try these fixes:

• Use DSE's built-in Spark cluster, not standalone Apache Spark: DSE Spark is customized to work with Cassandra, so a vanilla Apache Spark cluster won't have the right dependencies or configurations to communicate properly with DSE. Make sure the spark://127.0.0.1:7077 master is part of your DSE cluster (start it with dse spark start if needed).
• Ensure Worker nodes can reach your IDE's Driver: If you're running the IDE on your local machine, the Spark Worker might not be able to resolve 127.0.0.1 to your actual machine. Set the driver's explicit host address in your SparkSession config:

  .config("spark.driver.host", "your-local-ip") // e.g., 192.168.1.100, not 127.0.0.1
  

• Check Worker logs and classpath: Look at the DSE Worker logs (usually in /var/log/dse/spark/worker.log) to see if there are missing dependencies or connection errors. Ensure the Worker nodes have access to the DSE Spark-Cassandra connector JARs.
• Increase resource allocations: The error can also happen if the Driver/Executor doesn't have enough memory. Add these configs to your SparkSession:

  .config("spark.driver.memory", "1g")
  .config("spark.executor.memory", "2g")
  

• Test with dse spark-shell first: Before trying from IDE, run your query in dse spark-shell connected to the cluster to confirm the cluster itself is working correctly. If that works, the issue is definitely with your IDE's configuration.

内容的提问来源于stack exchange，提问作者Chinmay R