IntelliJ中Scala项目需添加哪些Apache Spark依赖？含GraphX等组件

Hey there! Let's break this down clearly, whether you're using sbt (the most common build tool for Scala/Spark projects) or Maven in your IntelliJ setup.

General Spark Dependencies (For Your First Question) #

At minimum, you'll start with Spark Core, then add any additional modules you need (like SQL, GraphX, Streaming, etc.). Just make sure your Spark version is compatible with your Scala version—for example, Spark 3.5.0 works smoothly with Scala 2.12.x or 2.13.x.

Exact Dependencies for Your Imports (Second Question) #

Your specific imports map to three key Spark modules: spark-core (for SparkConf), spark-graphx (for GraphLoader), and spark-sql (for SparkSession). Here's how to add each:

For sbt Projects #

Drop these lines into your build.sbt file:

// Spark Core (required for SparkConf)
libraryDependencies += "org.apache.spark" %% "spark-core" % "3.5.0" % Provided

// Spark GraphX (required for GraphLoader)
libraryDependencies += "org.apache.spark" %% "spark-graphx" % "3.5.0" % Provided

// Spark SQL (required for SparkSession)
libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.5.0" % Provided

• The %% automatically appends your project's Scala version to the artifact ID (so you don't have to manually write spark-core_2.12).
• The % Provided flag tells sbt not to package these dependencies into your final JAR—Spark clusters already have these libraries pre-installed, so this avoids version conflicts.

For Maven Projects #

Add these dependencies to your pom.xml file (replace 2.12 with 2.13 if that's your Scala version):

<!-- Spark Core (for SparkConf) -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.12</artifactId>
    <version>3.5.0</version>
    <scope>provided</scope>
</dependency>

<!-- Spark GraphX (for GraphLoader) -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-graphx_2.12</artifactId>
    <version>3.5.0</version>
    <scope>provided</scope>
</dependency>

<!-- Spark SQL (for SparkSession) -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.12</artifactId>
    <version>3.5.0</version>
    <scope>provided</scope>
</dependency>

• <scope>provided</scope> does the same thing as the sbt flag—keeps your JAR lean and avoids duplicate library conflicts on the cluster.

Quick Tips #

• If you're running your app locally (not on a cluster), you can remove the Provided/<scope>provided</scope> tag so IntelliJ includes the dependencies for local testing.
• Double-check that your IntelliJ Scala SDK matches the version specified in your build file (e.g., if you're using Scala 2.12, set your project SDK to 2.12.x).

内容的提问来源于stack exchange，提问作者zzmay