Spark库与CosmosDB库依赖冲突无法解决，请求技术协助

I’ve run into this exact headache before—Spark and CosmosDB connectors often clash over transitive dependencies like Azure Storage, Jackson, or Guava. Let’s walk through how to fix this step by step.

First, let’s pinpoint exactly what’s conflicting. Run this sbt command to get a full view of your dependency tree:

sbt dependencyTree

Look for lines marked with [conflict]—this will tell you which library and version is causing the problem (most likely azure-storage or a Jackson library, since both Spark and CosmosDB rely heavily on those).

1. Exclude Conflicting Transitive Dependencies #

Once you know the culprit, exclude it from the CosmosDB connector. I’ve added the common CosmosDB connector for Spark 2.3.x (since you didn’t include it in your snippet) and adjusted the dependencies to avoid clashes:

name := "myApp"
version := "1.0"
scalaVersion := "2.11.8"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "2.3.0",
  "org.apache.spark" %% "spark-sql" % "2.3.0",
  "org.apache.spark" %% "spark-streaming" % "2.3.0",
  "org.apache.spark" %% "spark-mllib" % "2.3.0",
  "com.microsoft.azure" % "azure-storage" % "2.0.0",
  "org.apache.hadoop" % "hadoop-azure" % "2.7.3", // Spark 2.3.0 uses Hadoop 2.7.3, match this version
  // CosmosDB connector with exclusions for conflicting libraries
  "com.microsoft.azure.cosmosdb" % "azure-cosmosdb-spark_2.11" % "1.2.2" excludeAll(
    ExclusionRule(organization = "com.microsoft.azure", name = "azure-storage"),
    ExclusionRule(organization = "com.fasterxml.jackson.core")
  )
)

The excludeAll clause tells sbt to ignore the conflicting versions that the CosmosDB connector tries to pull in, and instead use the ones you’ve specified for Spark.

2. Force Specific Versions If Exclusions Aren’t Enough #

If exclusions don’t resolve the conflict, you can force sbt to use a single version of the problematic library across all dependencies. Add this to your build.sbt:

dependencyOverrides ++= Seq(
  "com.microsoft.azure" % "azure-storage" % "2.0.0",
  "com.fasterxml.jackson.core" % "jackson-databind" % "2.6.7.1" // This is the version Spark 2.3.0 uses
)

This ensures every dependency in your project uses these versions, no matter what their transitive dependencies specify.

3. Double-Check Compatibility #

Make sure you’re using a CosmosDB connector version built for Spark 2.3.x and Scala 2.11. For your setup, azure-cosmosdb-spark_2.11:1.2.2 is the right pick—using a connector for a newer Spark version will cause all sorts of mismatches.

Quick Final Checks #

• After making changes, run sbt clean compile to clear cached dependencies and rebuild from scratch.
• If you’re still stuck, share the full output of sbt dependencyTree—that will let us spot the exact conflict point.

内容的提问来源于stack exchange，提问作者Tom