SBT配置异常：Scala+Spark项目无法引入log4j与Spark依赖

Hey there, let's get your dependency problem sorted out! It looks like your build.sbt file was cut off right at the libraryDependencies line — that's almost certainly why your project can't locate the Spark and Log4j classes. Here's a step-by-step fix:

1. Complete Your build.sbt Configuration #

Since you're using Scala 2.12.4, you need a Spark version compatible with this Scala release (Spark 3.x series supports Scala 2.12). Here's a full, working build.sbt for your project:

name := "SampleLearning"
version := "0.1"
scalaVersion := "2.12.4"

// Spark & Log4j dependencies (matched to Scala 2.12)
libraryDependencies ++= Seq(
  // Spark Core - core Spark functionality
  "org.apache.spark" %% "spark-core" % "3.3.4" % Provided,
  // Spark SQL (include if you plan to use SQL APIs)
  "org.apache.spark" %% "spark-sql" % "3.3.4" % Provided,
  // Log4j 1.2 (matches Spark's default logging setup)
  "log4j" % "log4j" % "1.2.17" % Provided,
  // SLF4J-to-Log4j bridge (required for Spark's logging integration)
  "org.slf4j" % "slf4j-log4j12" % "1.7.36" % Provided
)

Key notes on these dependencies: #

• The %% operator automatically appends your Scala version to the artifact ID (so spark-core becomes spark-core_2.12 for your setup, avoiding version mismatches)
• The Provided scope tells SBT these dependencies will be supplied by your Spark cluster at runtime — this prevents packaging duplicate libraries into your JAR and avoids conflict errors
• We include the SLF4J bridge because Spark uses SLF4J as its logging facade, and it needs this bridge to connect to Log4j

2. Troubleshooting If Issues Linger #

• Clean and rebuild: Run sbt clean compile in your terminal to clear cached dependencies and force a fresh build
• Sync your IDE: If using IntelliJ or another IDE, trigger a reimport of your SBT project (look for a "Refresh SBT Projects" button) — sometimes IDEs don't pick up dependency changes automatically
• Check network access: Ensure your machine can reach Maven Central (if behind a corporate proxy, you may need to add proxy settings to your ~/.sbt/repositories file)

3. Quick Validation of Your Code #

Your import statements are already correct for the dependencies we added:

package com.sparksql.count
import org.apache.log4j.Level
import org.apache.log4j.Logger
import org.apache.spark.SparkConf
import org.apache.spark._

Once you update your build.sbt and run a clean compile, those "cannot find symbol" errors should vanish!

内容的提问来源于stack exchange，提问作者Adam