在Hadoop环境运行Spark时遇Spark-submit报错：无法从JAR文件加载主类，如何解决？

Hey there, I’ve run into this exact "Cannot load main class from JAR file" error plenty of times when submitting Spark apps to Hadoop clusters. Let’s break down the most common fixes, starting with the easiest checks first:

• First things first: make sure the JAR path you’re using in spark-submit is correct. If you’re using a relative path, double-check your current working directory (run pwd to confirm). If the JAR is on HDFS, use the full HDFS URI like hdfs://<namenode-host>:<port>/path/to/your-app.jar—Spark won’t automatically look in HDFS without the prefix.
• Verify the file exists with ls /local/path/to/your.jar (local) or hadoop fs -ls /hdfs/path/to/your.jar (HDFS).

• You have two ways to tell Spark which class to run:
  1. Use the --class flag in your spark-submit command:

     spark-submit --class com.yourteam.YourMainClass --master yarn /path/to/your-app.jar
     

  2. Define the Main-Class attribute in your JAR’s MANIFEST.MF file (so you don’t need --class every time).
• Critical note: Java class names are case-sensitive, and you need the full qualified name (including package structure). Don’t just use YourMainClass unless it’s in the default package.
• You can check if the class exists in your JAR with:

  jar tf your-app.jar | grep YourMainClass
  

  This will show you the exact path to the class file—make sure it matches the full qualified name you’re using.

• A broken build is a frequent culprit. If you’re using Maven or Gradle, make sure you’re packaging the main class and all necessary dependencies correctly:
  • For Maven, use the maven-shade-plugin or maven-assembly-plugin to create an executable "fat JAR" (especially if your app has external dependencies). Here’s a quick shade plugin config snippet to set the main class:

    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-shade-plugin</artifactId>
      <version>3.4.1</version>
      <executions>
        <execution>
          <phase>package</phase>
          <goals>
            <goal>shade</goal>
          </goals>
          <configuration>
            <transformers>
              <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                <mainClass>com.yourteam.YourMainClass</mainClass>
              </transformer>
            </transformers>
          </configuration>
        </execution>
      </executions>
    </plugin>
    

• After building, extract the JAR with jar xf your-app.jar and check two things:
  1. The main class file exists in the correct package directory.
  2. The META-INF/MANIFEST.MF file has a line like Main-Class: com.yourteam.YourMainClass.

• If the JAR is on the local filesystem, the user running Spark (usually the yarn user on Hadoop clusters) needs read access. Run chmod +r your-app.jar to grant read permissions, or adjust the file’s group ownership if needed.
• For HDFS JARs, set proper permissions with:

  hadoop fs -chmod 644 /hdfs/path/to/your-app.jar
  

  Also ensure the Spark execution user has access to the parent directories in HDFS.

• It sounds silly, but typos are one of the most common causes. Double-check:
  • You’re using --class (not -class or a misspelled variant).
  • The main class name has no extra spaces, missing dots, or incorrect capitalization.
  • The JAR path has no typos (especially easy with long HDFS paths).

• While less common, a version mismatch between your Spark build and the Hadoop cluster can cause class loading issues. Make sure you’ve compiled or downloaded Spark for your cluster’s Hadoop version (e.g., Spark 3.3.x works best with Hadoop 3.2+).

内容的提问来源于stack exchange，提问作者katcsc