Scala SBT Assembly构建Kafka Streaming胖Jar报错求助（SBT 1.0.3）

Hey there! I’ve helped quite a few developers work through SBT assembly issues when building fat jars for Kafka Streams and Cassandra projects—let’s break down how to fix this and get your build working smoothly.

First: Verify Your plugins.sbt Configuration #

The sbt-assembly plugin needs to be correctly added to your project, and the version has to be compatible with SBT 1.0.3. Here’s the right setup for your project/plugins.sbt:

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.10") // This version works seamlessly with SBT 1.x

Double-check for typos here—missing or mismatched plugin versions are one of the most common causes of assembly failures.

Second: Fix build.sbt Key Configurations #

Most assembly errors stem from dependency conflicts (Kafka Streams and Cassandra share common dependencies like Netty or SLF4J) or missing main class declarations. Here’s a tailored build.sbt example for your use case:

name := "kafka-streaming-cassandra"
version := "1.0"
scalaVersion := "2.11.12" // Match this to your Kafka Streams version! Kafka 2.x uses Scala 2.11, 3.x uses 2.12/2.13

// Core project dependencies
libraryDependencies += "org.apache.kafka" % "kafka-streams" % "2.0.1"
libraryDependencies += "com.datastax.cassandra" % "cassandra-driver-core" % "3.7.2"

// Critical: Specify your application's main class (replace with your actual class path)
mainClass in assembly := Some("com.yourcompany.streaming.YourKafkaStreamApp")

// Resolve dependency conflicts with custom merge strategies
assemblyMergeStrategy in assembly := {
  // Handle META-INF files to avoid duplication errors
  case PathList("META-INF", xs @ _*) =>
    xs match {
      case ("MANIFEST.MF" :: Nil) => MergeStrategy.discard // Skip duplicate manifest files
      case ("services" :: _) => MergeStrategy.filterDistinctLines // Merge service files without duplicates
      case _ => MergeStrategy.first // Pick the first occurrence of other META-INF files
    }
  // Resolve conflicts in libraries shared by Kafka and Cassandra
  case PathList("io", "netty", xs @ _*) => MergeStrategy.first
  case PathList("org", "slf4j", xs @ _*) => MergeStrategy.first
  case PathList("scala", "reflect", "reflectasm", xs @ _*) => MergeStrategy.first
  // Fall back to the default strategy for all other files
  case x =>
    val oldStrategy = (assemblyMergeStrategy in assembly).value
    oldStrategy(x)
}

// Optional: Mark dependencies as "Provided" if they're available in your runtime environment
// (e.g., if your Kafka cluster already has Kafka Streams libraries installed)
libraryDependencies ++= Seq(
  "org.apache.kafka" % "kafka-streams" % "2.0.1" % Provided,
  "com.datastax.cassandra" % "cassandra-driver-core" % "3.7.2" % Provided
)

Step-by-Step Troubleshooting Tips #

1. Align Scala & Kafka Versions: If you’re using a newer Kafka version (like 3.x), switch your scalaVersion to 2.12 or 2.13—mismatched versions cause hidden dependency conflicts that break assembly.
2. Target Specific Duplicate Errors: If you get a deduplicate: different file contents found error, add the path from the error message to the merge strategy. For example, if it’s about a com/google/common file, add:

   case PathList("com", "google", "common", xs @ _*) => MergeStrategy.first
   

3. Clean Before Building: Run sbt clean assembly instead of just sbt assembly—cached build artifacts can sometimes cause unexpected, hard-to-debug issues.

If you hit a specific error message (like missing classes or unresolved dependencies), share that detail and we can refine the fix further!

内容的提问来源于stack exchange，提问作者Gnana