sbt构建Scala Spark项目不升级版本修复Log4j漏洞的方法

项目基础环境 #

• Scala版本：2.11.8
• Spark版本：2.2.0
  原有依赖配置：

libraryDependencies ++=
  Seq(
    "org.apache.spark" %% "spark-core" % "2.2.0" ,
    "org.apache.spark" %% "spark-sql" % "2.2.0"  ,
    "org.apache.spark" %% "spark-hive" % "2.2.0"
  )

原有Scala版本配置：

ThisBuild / scalaVersion := "2.11.8"

现存问题 #

1. 构建时抛出依赖解析错误，报错信息如下：

[error] sbt.librarymanagement.ResolveException: unresolved dependency: org.apache.logging.log4j#log4j-api;2.11.1: Resolution failed several times for dependency: org.apache.logging.log4j#log4j-api;2.11.1 {compile=[compile(*), master(*)], runtime=[runtime(*)]}::
[error]     typesafe-ivy-releases: unable to get resource for org.apache.logging.log4j#log4j-api;2.11.1: res=https://repo.typesafe.com/typesafe/ivy-releases/org.apache.logging.log4j/log4j-api/2.11.1/ivys/ivy.xml: java.io.IOException: Unexpected response code for CONNECT: 403
[error]     sbt-plugin-releases: unable to get resource for org.apache.logging.log4j#log4j-api;2.11.1: res=https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/org.apache.logging.log4j/log4j-api/2.11.1/ivys/ivy.xml: java.io.IOException: Unexpected response code for CONNECT: 403

2. 安全规范要求移除所有存在漏洞的旧版本log4j-core jar包，直接移除后所有通过传递依赖引入Log4j的项目均构建失败。
3. 已尝试的两种方案均未生效：
   • 单独通过dependencyOverrides指定log4j-core版本为2.17.2
   • 对Spark依赖配置excludeAll排除规则
     要求在不升级Scala、Spark版本的前提下，强制构建工具不拉取存在漏洞的旧版log4j-core，替换使用无漏洞的2.17.2版本。

可行配置方案 #

第一步：修复依赖拉取403问题 #

原有配置中默认的typesafe-ivy、sbt-plugin-releases仓库已停止维护大量第三方依赖，直接替换为可用的Maven中央仓库即可解决拉包403问题，在build.sbt中添加如下配置：

// 覆盖默认解析仓库，移除失效的老ivy仓库
externalResolvers := Seq(
  "Maven Central" at "https://repo1.maven.org/maven2/",
  Resolver.mavenLocal
)

第二步：全局统一覆盖所有Log4j相关模块版本 #

仅覆盖log4j-core单包会因为Log4j模块间版本校验不通过，导致构建时仍拉取旧版本传递依赖，必须将Log4j核心模块、桥接模块版本统一覆盖为2.17.2：

dependencyOverrides ++= Seq(
  "org.apache.logging.log4j" % "log4j-api" % "2.17.2",
  "org.apache.logging.log4j" % "log4j-core" % "2.17.2",
  "org.apache.logging.log4j" % "log4j-slf4j-impl" % "2.17.2",
  // 引入log4j1.x桥接包，兼容Spark 2.2.0内部的旧版日志API调用
  "org.apache.logging.log4j" % "log4j-1.2-api" % "2.17.2"
)

第三步：精准排除Spark依赖中的旧版日志包 #

对所有Spark依赖精准排除旧版log4j、slf4j-log4j12绑定包，避免旧版漏洞包被传递引入：

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "2.2.0" excludeAll(
    ExclusionRule(organization = "log4j"),
    ExclusionRule(organization = "org.slf4j", name = "slf4j-log4j12"),
    ExclusionRule(organization = "org.apache.logging.log4j", name = "log4j-core")
  ),
  "org.apache.spark" %% "spark-sql" % "2.2.0" excludeAll(
    ExclusionRule(organization = "log4j"),
    ExclusionRule(organization = "org.slf4j", name = "slf4j-log4j12"),
    ExclusionRule(organization = "org.apache.logging.log4j", name = "log4j-core")
  ),
  "org.apache.spark" %% "spark-hive" % "2.2.0" excludeAll(
    ExclusionRule(organization = "log4j"),
    ExclusionRule(organization = "org.slf4j", name = "slf4j-log4j12"),
    ExclusionRule(organization = "org.apache.logging.log4j", name = "log4j-core")
  )
)

第四步：验证配置生效 #

执行sbt dependencyTree命令查看全量依赖树，搜索所有log4j相关条目，确认：

• 所有org.apache.logging.log4j组下的依赖版本均为2.17.2
• 不存在1.x版本的log4j包
• 不存在版本低于2.17.2的log4j-core包

注意事项 #

• Log4j的api、core模块版本必须严格一致，否则会出现运行时类找不到、方法签名不匹配的错误
• 不要使用无差别excludeAll排除所有传递依赖，会导致Spark依赖的其他必要包丢失，精准排除日志相关包即可
• log4j-1.2-api桥接包必须引入，否则Spark内部调用旧版log4j 1.x API时会抛出类不存在异常

内容的提问来源于stack exchange，提问作者Sangeeta