Spark Submit运行打包Jar报java.lang.NoSuchMethodError问题求助

Hey there, that NoSuchMethodError is almost always a sign of dependency version conflicts or mismatches between your local development setup and the Spark cluster environment. Let's walk through the most likely fixes:

1. Align Spark and Hadoop Version Compatibility #

The method org.apache.hadoop.conf.Configuration.reloadExistingConfigurations() was introduced in Hadoop 3.1+. If your cluster runs an older Hadoop version (like 2.x), but your project's pom.xml pulls in a newer Hadoop dependency, the cluster's runtime won't recognize this method.

• First, check your Spark cluster's Hadoop version by running hadoop version on cluster nodes.
• Match your project's Hadoop dependencies in pom.xml to the cluster's version. For example, if the cluster uses Hadoop 2.7.3, update your pom like this:

  <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-common</artifactId>
      <version>2.7.3</version>
      <scope>provided</scope>
  </dependency>
  

2. Mark Core Spark/Hadoop Dependencies as provided #

When running locally in IntelliJ, your app uses the dependencies in your project. But Spark clusters already include core Spark and Hadoop libraries—packaging these into your JAR causes clashes with the cluster's existing jars.

• Update your pom.xml to set core Spark and Hadoop dependencies to provided scope:

  <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.12</artifactId>
      <version>your-spark-version</version>
      <scope>provided</scope>
  </dependency>
  <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.12</artifactId>
      <version>your-spark-version</version>
      <scope>provided</scope>
  </dependency>
  

  This tells Maven not to include these jars in your final package, letting the cluster's native libraries take precedence.

3. Use Maven Shade Plugin for Safe Fat Jar Packaging #

If you need to include custom dependencies (not part of Spark/Hadoop core), use the maven-shade-plugin instead of a standard mvn package to avoid dependency conflicts. This plugin can relocate conflicting classes and merge jars safely.

Add this to your pom.xml's <build><plugins> section:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.2.4</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <relocations>
                    <relocation>
                        <pattern>com.google</pattern>
                        <shadedPattern>shaded.com.google</shadedPattern>
                    </relocation>
                </relocations>
                <filters>
                    <filter>
                        <artifact>*:*</artifact>
                        <excludes>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                        </excludes>
                    </filter>
                </filters>
            </configuration>
        </execution>
    </executions>
</plugin>

Then build with mvn clean shade:shade instead of mvn package to generate a properly shaded JAR.

Why This Works Locally But Not On Cluster? #

Your IntelliJ setup uses the exact dependencies you've defined, and your local environment probably has a matching Hadoop version. The cluster, however, has pre-installed libraries that don't align with the ones you packaged, leading to the method-not-found error.

内容的提问来源于stack exchange，提问作者Omkar