Java/Scala/Kotlin生态中如何可靠重打包依赖规避版本冲突？

这是个老问题，但至今没得到妥善解决，根源要从Apache Spark的开发细节说起。

在Spark 1.x和2.x的发布阶段，团队发现核心依赖Apache Hive 1.x带来了大量过时的传递依赖，和YARN/HDFS部署环境极易冲突。由于当时没有足够资源推行单仓原则（确保依赖树中每个库仅存在一个版本），团队选择对Apache Hive进行硬分叉：修改所有源码中org.apache.hive的包名为org.spark-project.hive，重新编译发布。

这种手动重打包的方式缺点很明显：无法跟进Hive社区的更新，要么放弃新特性，要么投入大量精力同步代码；此外还存在安全风险——未签名的迁移Jar包可能被替换。所以Spark 3.0之后，团队有了足够资源升级到依赖更合理的官方Hive 2.x，这个手动迁移项目就被终止了。

自动化重打包的尝试与问题 #

原本以为Spark 2.0发布5年后，编译工具的改进能让这类流程自动化。Maven Shade Plugin和Gradle Shadow Plugin就是专为重定位依赖包设计的，理论上可以直接从标准依赖生成迁移后的字节码，但实际测试却失败了。

测试项目包含两个重打包子项目：

Maven子项目配置 #

用Maven Shade Plugin将json4s重定位到repacked.test1.org.json4s：

<dependencies>
  <dependency>
    <groupId>org.json4s</groupId>
    <artifactId>json4s-jackson_${vs.scalaBinaryV}</artifactId>
    <version>4.0.4</version>
  </dependency>
</dependencies>

<build>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-shade-plugin</artifactId>
      <version>3.2.4</version>
      <executions>
        <execution>
          <phase>package</phase>
          <goals>
            <goal>shade</goal>
          </goals>
          <configuration>
            <createDependencyReducedPom>true</createDependencyReducedPom>
            <dependencyReducedPomLocation>${project.build.directory}/dependency-reduced-pom.xml</dependencyReducedPomLocation>

            <keepDependenciesWithProvidedScope>false</keepDependenciesWithProvidedScope>
            <promoteTransitiveDependencies>false</promoteTransitiveDependencies>

            <relocations>
              <relocation>
                <pattern>org.json4s</pattern>
                <shadedPattern>repacked.test1.org.json4s</shadedPattern>
              </relocation>
            </relocations>

            <filters>
              <filter>
                <artifact>*:*</artifact>
                <excludes>
                  <exclude>META-INF/*.SF</exclude>
                  <exclude>META-INF/*.DSA</exclude>
                </excludes>
              </filter>
            </filters>
          </configuration>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>

Gradle子项目配置 #

用Shadow Plugin将json4s重定位到repacked.test2.org.json4s：

dependencies {
    api("org.json4s:json4s-jackson_${vs.scalaBinaryV}:4.0.4")
}

tasks {
    shadowJar {
        exclude("META-INF/*.SF")
        exclude("META-INF/*.DSA")
        relocate("org.json4s", "repacked.test2.org.json4s")
    }
}

测试代码与报错 #

第三个Gradle项目引入上述两个重打包依赖，用Scala尝试访问重定位后的类：

dependencies {
    api(project(":repack:gradle", configuration = "shadow"))
    api("com.tribbloids.autoshade:repack-maven:0.0.1")
}

class Json4sTest {
  classOf[repacked.test1.org.json4s.Formats]
  classOf[repacked.test2.org.json4s.Formats]
}

编译直接失败，报错如下：

[Error] /home/peng/git-proto/autoshade/main/src/main/scala/com/tribbloids/spookystuff/Json4sTest.scala:7:11: Symbol 'term org.json4s' is missing from the classpath.
This symbol is required by ' <none>'.
Make sure that term json4s is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
A full rebuild may help if 'package.class' was compiled against an incompatible version of org.
[Error] /home/peng/git-proto/autoshade/main/src/main/scala/com/tribbloids/spookystuff/Json4sTest.scala:7:28: type Formats is not a member of package repacked.test1.org.json4s
[Error] /home/peng/git-proto/autoshade/main/src/main/scala/com/tribbloids/spookystuff/Json4sTest.scala:10:11: Symbol 'term org.json4s' is missing from the classpath.
This symbol is required by ' <none>'.
Make sure that term json4s is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
A full rebuild may help if 'package.class' was compiled against an incompatible version of org.
[Error] /home/peng/git-proto/autoshade/main/src/main/scala/com/tribbloids/spookystuff/Json4sTest.scala:10:28: type Formats is not a member of package repacked.test2.org.json4s

从报错信息看，包迁移明显不完整且不一致——如果只是引用不存在的类，不会出现前两条关于org.json4s缺失的错误，这说明自动化工具的重定位逻辑存在漏洞，和Spark团队手动修改源码的效果完全没法比。

为什么这么简单的自动化任务都做不到？在Maven或Gradle中需要额外执行哪些步骤才能让重打包正常工作？

内容的提问来源于stack exchange，提问作者tribbloid