Spark 2.2.0执行spark-submit时提示Class Not Found问题求助

Let's break down the possible causes and fixes for your Class Not Found error when running spark-submit with Spark 2.2.0. I'll walk through the most common issues and actionable solutions:

1. Verify Jar Paths Are Correct #

The --jars parameter in your command uses relative filenames (spark-cassandra-connector_2.11-2.0.5.jar,cms-api-4.1.0-SNAPSHOT.jar), which only works if those jars exist in the exact directory where you run spark-submit.

Fix: #

Use absolute paths for your jars to avoid path resolution issues:

--jars /opt/cmsgraph/jars/spark-cassandra-connector_2.11-2.0.5.jar,/opt/cmsgraph/jars/cms-api-4.1.0-SNAPSHOT.jar

Or, if your cluster uses HDFS, host the jars there so all executor nodes can access them:

--jars hdfs://your-cluster-url/path/to/spark-cassandra-connector_2.11-2.0.5.jar,hdfs://your-cluster-url/path/to/cms-api-4.1.0-SNAPSHOT.jar

2. Check Dependency Version Compatibility #

Your spark-cassandra-connector version (2.0.5) is technically compatible with Spark 2.2.0 (connector 2.0.x maps to Spark 2.2.x), but double-check:

• The jar is compiled for Scala 2.11 (your jar name has _2.11, which matches Spark 2.2.0's default Scala version)
• If your application code uses Scala, ensure it was compiled with Scala 2.11 (mismatched Scala versions often cause hidden Class Not Found errors)

3. Resolve Missing Transitive Dependencies #

The cms-api-4.1.0-SNAPSHOT.jar likely depends on other libraries that aren't included in your --jars list. Spark won't automatically pull in these transitive dependencies unless you tell it to.

Fixes: #

Option A: Build a Fat Jar

Use Maven's maven-assembly-plugin or SBT's sbt-assembly plugin to package your application and all its dependencies into a single "fat jar". This eliminates missing dependency issues entirely—you only need to specify this fat jar in your spark-submit command (no --jars needed for your app's dependencies).

Option B: Use --packages for Managed Dependencies

If spark-cassandra-connector is available in a Maven repository, replace --jars with --packages to auto-pull transitive dependencies:

--packages com.datastax.spark:spark-cassandra-connector_2.11:2.0.5

For your internal cms-api jar, add your company's Maven repository:

--repositories http://your-internal-maven-repo/repository/public/
--packages com.datastax.spark:spark-cassandra-connector_2.11:2.0.5,com.yourcompany:cms-api:4.1.0-SNAPSHOT

4. Confirm Executor Node Access #

If your jars are stored locally on the driver node only, executor nodes won't be able to access them. Ensure:

• Jars are placed in the same absolute path on all cluster nodes, or
• Jars are hosted on a shared filesystem (like HDFS) that all nodes can reach

Example Corrected Command #

Here's how your command might look with absolute paths and proper structure:

/opt/cmsgraph/spark/default/bin/spark-submit -v \
--driver-java-options -Djava.io.tmpdir=/opt/cmsgraph/temp \
--conf spark.cassandra.connection.timeout_ms=60000 \
--conf spark.cassandra.input.fetch.size_in_rows=1 \
--conf spark.cassandra.input.split.size_in_mb=8 \
--conf spark.cassandra.input.reads_per_sec=1 \
--conf spark.cassandra.output.consistency.level=LOCAL_ONE \
--jars /opt/cmsgraph/jars/spark-cassandra-connector_2.11-2.0.5.jar,/opt/cmsgraph/jars/cms-api-4.1.0-SNAPSHOT.jar \
--master spark://your-master-host:7077 \
--class com.your.package.YourMainApplicationClass \
/path/to/your-application.jar

Pro tip: Always check the exact class name in the error message—it will tell you whether the missing class comes from the Cassandra connector, your cms-api, or a hidden transitive dependency.

内容的提问来源于stack exchange，提问作者Himanshu Gangwar