Spark Streaming 2.3.0与MaxMind依赖Jackson版本冲突解决问询

问题背景 #

你在使用Spark Streaming 2.3.0（内置Jackson 2.6.7）时，引入了依赖Jackson 2.9.5的MaxMind库来解析IP地理信息。独立测试代码可以正常运行，但在Spark Streaming中通过广播DatabaseReader调用getGeoFromIP方法时，遇到了Jackson版本冲突导致的AWS SDK CBOR错误：

Exception in thread "main" com.amazonaws.SdkClientException: Unable to marshall request to JSON: Jackson jackson-core/jackson-dataformat-cbor incompatible library version detected. You have two possible resolutions:

1. Ensure the com.fasterxml.jackson.core:jackson-core & com.fasterxml.jackson.dataformat:jackson-dataformat-cbor libraries on your classpath have the same version number
2. Disable CBOR wire-protocol by passing the -Dcom.amazonaws.sdk.disableCbor property or setting the AWS_CBOR_DISABLE environment variable (warning this may affect performance)

解决方案：强制统一Jackson版本为2.9.5 #

要让项目强制使用Jackson 2.9.5覆盖Spark自带的2.6.7，你需要修改build.sbt，排除Spark依赖中的Jackson模块，然后显式引入2.9.5版本的Jackson核心包。

修改后的build.sbt配置

name := "scala_spark_stream_metrices"
version := "1.0"
scalaVersion := "2.11.8"

// 排除Spark各模块依赖中的Jackson相关包
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.0" excludeAll(
  ExclusionRule(organization = "com.fasterxml.jackson.core"),
  ExclusionRule(organization = "com.fasterxml.jackson.module"),
  ExclusionRule(organization = "com.fasterxml.jackson.datatype")
)

libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.3.0" excludeAll(
  ExclusionRule(organization = "com.fasterxml.jackson.core"),
  ExclusionRule(organization = "com.fasterxml.jackson.module"),
  ExclusionRule(organization = "com.fasterxml.jackson.datatype")
)

libraryDependencies += "org.apache.spark" %% "spark-streaming-kinesis-asl" % "2.3.0" excludeAll(
  ExclusionRule(organization = "com.fasterxml.jackson.core"),
  ExclusionRule(organization = "com.fasterxml.jackson.module"),
  ExclusionRule(organization = "com.fasterxml.jackson.datatype")
)

libraryDependencies += "org.apache.spark" %% "spark-streaming" % "2.3.0" excludeAll(
  ExclusionRule(organization = "com.fasterxml.jackson.core"),
  ExclusionRule(organization = "com.fasterxml.jackson.module"),
  ExclusionRule(organization = "com.fasterxml.jackson.datatype")
)

// 保留原有其他依赖
libraryDependencies += "org.elasticsearch" % "elasticsearch-hadoop" % "6.2.3"
libraryDependencies += "com.maxmind.geoip2" % "geoip2" % "2.12.0"

// 显式引入Jackson 2.9.5版本的核心依赖，确保全项目统一
libraryDependencies ++= Seq(
  "com.fasterxml.jackson.core" % "jackson-core" % "2.9.5",
  "com.fasterxml.jackson.core" % "jackson-databind" % "2.9.5",
  "com.fasterxml.jackson.core" % "jackson-annotations" % "2.9.5",
  "com.fasterxml.jackson.dataformat" % "jackson-dataformat-cbor" % "2.9.5"
)

配置说明 #

1. 排除Spark自带Jackson：通过excludeAll规则移除Spark各模块依赖的Jackson相关包，彻底避免低版本Jackson进入项目类路径。
2. 统一引入高版本Jackson：添加Jackson核心的三个基础包（core、databind、annotations）以及AWS SDK需要的jackson-dataformat-cbor，确保MaxMind、AWS SDK和Spark都使用同一版本的Jackson，从根源解决版本冲突。

临时替代方案（不推荐） #

如果暂时无法修改依赖配置，可以按照错误提示禁用CBOR协议，但这会影响AWS SDK的性能：

• 启动Spark应用时添加JVM参数：-Dcom.amazonaws.sdk.disableCbor=true
• 或者设置环境变量：export AWS_CBOR_DISABLE=true

内容的提问来源于stack exchange，提问作者ajaypr55