使用--jars选项时spark-submit报‘Missing application resource’失败的原因及解决

Hey there! I totally get the confusion when you're starting out with Spark—let's break down why you're seeing that error and fix it right away.

Why the Error Pops Up #

The spark-submit command needs two critical pieces to run your job successfully:

1. Dependency JARs (the ones you specified with --jars): These are extra libraries your application relies on.
2. Your core application resource: This is either a JAR file containing your job's main entry class, or a script (like .py or .scala) that defines your Spark job logic.

Right now, you've only told Spark about your dependency JAR, but you haven't pointed it to the actual application you want to execute. That's exactly why it throws the "Missing application resource" error.

How to Fix It #

Adjust your command based on the type of Spark application you're running:

Case 1: You have a compiled application JAR

If you've packaged your Spark job into a JAR (with a main class), add its path at the end of the command. If your JAR doesn't list the main class in its MANIFEST file, use the --class flag to specify it explicitly:

# Basic command (if main class is already in the JAR's MANIFEST)
./spark-submit --jars /home/parallels/Downloads/my_jar_name /path/to/your/spark_application.jar

# Specify main class if it's not in the MANIFEST
./spark-submit --jars /home/parallels/Downloads/my_jar_name --class com.yourteam.YourMainClass /path/to/your/spark_application.jar

Case 2: You're running a script (Python/Scala)

If you're using a script instead of a compiled JAR, just append the script's path to the command:

./spark-submit --jars /home/parallels/Downloads/my_jar_name /path/to/your/spark_job_script.py

Quick Extra Tip #

If you have multiple dependency JARs, separate them with commas (no spaces!) like this:

./spark-submit --jars jar1.jar,jar2.jar,/home/parallels/Downloads/my_jar_name /path/to/your/app.jar

内容的提问来源于stack exchange，提问作者Mike Wang