Scala开发Spark应用：是否需本地安装Spark？仅靠Maven依赖可行吗？

Great question—this is a common point of confusion when getting started with Spark and Maven. Let’s break it down based on what you’re trying to do:

1. Development & Local Code Execution #

You do NOT need to install Spark locally just to write, compile, or even run your Spark application in an IDE (like IntelliJ or Eclipse).

Maven handles all the heavy lifting here: when you add the correct Spark dependencies to your pom.xml, Maven will automatically download the required Spark JARs (and their dependencies) into your local Maven repository. These JARs include everything needed to run Spark in a local mode directly from your IDE.

Here’s a sample snippet of what those dependencies might look like (make sure the Scala version suffix matches your project’s Scala version—e.g., _2.12 for Scala 2.12):

<dependencies>
    <!-- Spark Core -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.12</artifactId>
        <version>3.5.0</version>
        <!-- Use 'provided' if you plan to submit to a cluster (more on that below) -->
        <!-- <scope>provided</scope> -->
    </dependency>
    <!-- Spark SQL (if your app uses it) -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.12</artifactId>
        <version>3.5.0</version>
        <!-- <scope>provided</scope> -->
    </dependency>
</dependencies>

If you’re running your app locally via your IDE, you can omit the <scope>provided</scope> tag so Maven includes the Spark JARs in your runtime classpath.

2. Submitting to a Remote Spark Cluster #

Again, you don’t need a local Spark installation to deploy your app to a remote cluster.

First, build a self-contained JAR of your application using Maven (typically with plugins like maven-assembly-plugin or maven-shade-plugin to handle dependencies). Then, you can use the spark-submit command that’s already installed on the cluster’s master node to submit your JAR—no local Spark setup required.

Note: When building for cluster deployment, it’s best to add <scope>provided</scope> to your Spark dependencies. This tells Maven not to include the Spark core JARs in your app’s JAR, since the cluster already has those libraries installed (avoiding redundant files and version conflicts).

3. Local Testing with spark-submit #

If you want to test your app using the official spark-submit command on your local machine (instead of running it through your IDE), then you will need to install a local Spark distribution. This is because spark-submit is a tool included with the Spark installation package—it’s not part of the Maven dependencies.

That said, this is optional. Most developers start by testing directly in their IDE before moving to spark-submit for more realistic local cluster testing.

内容的提问来源于stack exchange，提问作者user9683185