在Scala IDE中构建含Spark的Maven项目时遭遇执行异常

Hey there! Let’s work through this Maven build issue with your Spark + Scala project together. Since you skipped the archetype and set up a custom POM, here are the most common fixes that usually resolve these kinds of execution exceptions:

Spark and Scala have strict version matching rules—mismatches here are one of the top causes of build failures. For example:

• Spark 3.3.x pairs with Scala 2.12
• Spark 3.4+ supports Scala 2.13
• Older Spark 2.x versions rely on Scala 2.11

Make sure your POM’s <properties> and Spark dependencies align. Here’s a valid snippet to reference:

<properties>
    <scala.version>2.12.15</scala.version>
    <spark.version>3.3.2</spark.version>
    <scala.binary.version>2.12</scala.binary.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_${scala.binary.version}</artifactId>
        <version>${spark.version}</version>
    </dependency>
</dependencies>

Skipping the archetype often means the critical Scala Maven plugin is missing or misconfigured. Add this to your POM’s <build> section to handle Scala compilation properly:

<build>
    <plugins>
        <plugin>
            <groupId>net.alchim31.maven</groupId>
            <artifactId>scala-maven-plugin</artifactId>
            <version>4.6.2</version>
            <executions>
                <execution>
                    <goals>
                        <goal>compile</goal>
                        <goal>testCompile</goal>
                    </goals>
                    <configuration>
                        <scalaVersion>${scala.version}</scalaVersion>
                    </configuration>
                </execution>
            </executions>
        </plugin>
        <!-- Align Java compiler settings with Scala's requirements -->
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.11.0</version>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
        </plugin>
    </plugins>
</build>

Spark brings in a lot of transitive dependencies, and clashes with other libraries can break builds. Run this command to generate a dependency tree and spot conflicts:

mvn dependency:tree

Look for duplicate versions of libraries like Guava, Jackson, or Log4j. You can exclude conflicting transitive dependencies from your Spark dependencies like this:

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_${scala.binary.version}</artifactId>
    <version>${spark.version}</version>
    <exclusions>
        <exclusion>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
        </exclusion>
    </exclusions>
</dependency>

Stale build artifacts are a frequent culprit. Run these commands to wipe old files and rebuild from scratch:

mvn clean
mvn compile

In Scala IDE, you can also right-click your project → Maven → Update Project (check "Force Update of Snapshots/Releases") to refresh dependencies and sync the IDE with your POM.

Make sure Scala IDE is using the right tools:

• Go to Window → Preferences → Maven → Installations to confirm your Maven path is correct (not the embedded one if you have a local installation).
• Go to Window → Preferences → Scala → Installations to ensure the Scala version matches exactly what’s in your POM.

If none of these fix the issue, sharing the exact error message from your Maven build log would help zero in on the problem—specific exceptions like NoClassDefFoundError, compilation errors, or dependency resolution failures give crucial clues!

内容的提问来源于stack exchange，提问作者Nivedita Singh