解决SBT项目中Avro与Protobuf代码生成插件冲突问题

场景描述 #

原本通过sbt-avrohugger（2.0.0-RC19）实现从*.avsc、*.avdl文件生成Scala代码的功能正常。添加sbt-protoc（1.0.7）和ScalaPB compilerplugin（0.11.15）实现*.proto文件代码生成后，Protobuf代码生成正常，但Avro仅生成部分文件，导致测试编译失败。

配置代码 #

plugins.sbt #

addSbtPlugin("com.julianpeeters" % "sbt-avrohugger" % "2.0.0-RC19")
addSbtPlugin("com.thesamet" % "sbt-protoc" % "1.0.7")

libraryDependencies += "com.thesamet.scalapb" %% "compilerplugin" % "0.11.15"

build.sbt #

lazy val `job` = (project in file("."))
  .settings(avroHuggerSettings)
  .settings(protobufSettings)

lazy val avroHuggerSettings = Seq.concat(
  inConfig(IntegrationTest)(SbtAvrohugger.baseSettings),
  Seq(
    Compile / sourceGenerators += (Compile / avroScalaGenerate).taskValue,
    Test / sourceGenerators += (Test / avroScalaGenerate).taskValue,
    IntegrationTest / sourceGenerators += (IntegrationTest / avroScalaGenerate).taskValue,
    Compile / avroSourceDirectories := Seq(new java.io.File(s"${baseDirectory.value}/src/main/resources/avro")),
    Test / avroSourceDirectories := Seq(new java.io.File(s"${baseDirectory.value}/src/test/resources/avro")),
    IntegrationTest / avroSourceDirectories := Seq(new java.io.File(s"${baseDirectory.value}/src/it/resources/avro")),
    Compile / avroScalaCustomTypes := avroCustomTypes,
    Test / avroScalaCustomTypes := avroCustomTypes,
    IntegrationTest / avroScalaCustomTypes := avroCustomTypes
  )
)

lazy val protobufSettings = Seq(
  Compile / PB.targets := Seq(
    scalapb.gen() -> (Compile / sourceManaged).value
  )
)

文件路径说明：

• Avro文件：src/main/resources/avro、src/test/resources/avro、src/it/resources/avro
• Proto文件：src/main/protobuf
• 生成代码目录：target/scala-2.12/src_managed/main（分不同包）

构建日志片段 #

[info] Considering source directories ...resources/avro
[info] Compiling AVSC ... .avsc to .../target/scala-2.12/src_managed/main/compiled_avro
[info] Compiling AVSC ... .avsc to .../target/scala-2.12/src_managed/main/compiled_avro
[info] Compiling Avro IDL ... .avdl
[info] Compiling 4 protobuf files to .../target/scala-2.12/src_managed/main
[info] Compiling Avro IDL ... .avdl
[info] Compiling Avro IDL ... .avdl
[info] Compiling Avro IDL ... .avdl
[info] Compiling Avro IDL ... .avdl
[info] Compiling Avro IDL ... .avdl
[info] Compiling Avro IDL ... .avdl
[info] compiling 139 Scala sources to .../target/scala-2.12/classes
[info] done compiling
// same as above but for some test avro classes
// failing test output
[error] (Test / compileIncremental) Compilation failed
[error] Total time: 9 s, completed Jun 5, 2024, 4:11:10 PM

已尝试的方法 #

• 尝试调整插件运行顺序，定义自定义TaskKey控制执行顺序，无效果
• 试错法修改配置，未解决问题

解决方案 #

1. 显式设置任务依赖，确保Avro生成优先完成 #

SBT的sourceGenerators会并行执行任务，可能导致Avro生成未完成就开始编译或Protobuf生成。可以让Protobuf生成任务依赖于Avro生成任务，强制顺序执行：

// 修改protobufSettings，添加依赖
lazy val protobufSettings = Seq(
  Compile / PB.compileProtoc := (Compile / PB.compileProtoc).dependsOn(Compile / avroScalaGenerate).value,
  Test / PB.compileProtoc := (Test / PB.compileProtoc).dependsOn(Test / avroScalaGenerate).value,
  IntegrationTest / PB.compileProtoc := (IntegrationTest / PB.compileProtoc).dependsOn(IntegrationTest / avroScalaGenerate).value,
  Compile / PB.targets := Seq(
    scalapb.gen() -> (Compile / sourceManaged).value
  )
)

2. 分离Avro与Protobuf的代码生成目录 #

虽然两者生成在不同包，但共享同一sourceManaged根目录可能引发文件系统竞争。给Avro生成指定独立子目录：

// 修改avroHuggerSettings中的生成目录配置
lazy val avroHuggerSettings = Seq.concat(
  inConfig(IntegrationTest)(SbtAvrohugger.baseSettings),
  Seq(
    // 为每个配置设置独立的生成目录
    Compile / avroScalaGenerate := SbtAvrohugger.AvroScalaGenerateTask.generate(
      (Compile / avroSourceDirectories).value,
      (Compile / sourceManaged).value / "avro", // Avro生成到avro子目录
      (Compile / avroScalaCustomTypes).value,
      (Compile / avroScalaGenerator).value
    ),
    Test / avroScalaGenerate := SbtAvrohugger.AvroScalaGenerateTask.generate(
      (Test / avroSourceDirectories).value,
      (Test / sourceManaged).value / "avro",
      (Test / avroScalaCustomTypes).value,
      (Test / avroScalaGenerator).value
    ),
    IntegrationTest / avroScalaGenerate := SbtAvrohugger.AvroScalaGenerateTask.generate(
      (IntegrationTest / avroSourceDirectories).value,
      (IntegrationTest / sourceManaged).value / "avro",
      (IntegrationTest / avroScalaCustomTypes).value,
      (IntegrationTest / avroScalaGenerator).value
    ),
    // 其他原有配置不变
    Compile / sourceGenerators += (Compile / avroScalaGenerate).taskValue,
    Test / sourceGenerators += (Test / avroScalaGenerate).taskValue,
    IntegrationTest / sourceGenerators += (IntegrationTest / avroScalaGenerate).taskValue,
    Compile / avroSourceDirectories := Seq(new java.io.File(s"${baseDirectory.value}/src/main/resources/avro")),
    Test / avroSourceDirectories := Seq(new java.io.File(s"${baseDirectory.value}/src/test/resources/avro")),
    IntegrationTest / avroSourceDirectories := Seq(new java.io.File(s"${baseDirectory.value}/src/it/resources/avro")),
    Compile / avroScalaCustomTypes := avroCustomTypes,
    Test / avroScalaCustomTypes := avroCustomTypes,
    IntegrationTest / avroScalaCustomTypes := avroCustomTypes
  )
)

3. 清理缓存后重新构建 #

SBT的增量编译缓存可能导致任务未触发，执行以下命令彻底清理后重新构建：

sbt clean assembly

4. 确保Avro文件全量被扫描 #

检查Avro源目录是否包含所有子目录，添加includeFilter确保所有.avsc/.avdl文件被处理：

// 在avroHuggerSettings中添加
Compile / avroIncludeFilter := "*.avsc" || "*.avdl",
Test / avroIncludeFilter := "*.avsc" || "*.avdl",
IntegrationTest / avroIncludeFilter := "*.avsc" || "*.avdl"

内容的提问来源于stack exchange，提问作者mintas123