使用Terraform部署Databricks集群合并多个library属性块去重咨询

你可以通过Terraform的dynamic嵌套块功能批量生成library配置块，无需手动重复声明。注意：Databricks Terraform Provider不支持单library块内配置多个包，你期望的数组传入单pypi块的写法不符合provider的参数规范，必须通过动态生成多library块的方式实现。

基础实现（统一源的包批量配置） #

如果你的Python包、JAR包都使用统一的源，可以直接遍历包名列表生成配置：

variable "python_packages" {
  type    = list(string)
  default = ["conda", "pymongo", "dnspython"]
}

variable "jar_packages" {
  type    = list(string)
  default = ["etc1", "etc2"]
}

variable "pypi_repo" {
  type    = string
  default = "" # 填私有源地址，留空则用公共PyPI源
}

resource "databricks_cluster" "shared_autoscaling" {
  cluster_name            = "Shared Autoscaling"
  spark_version           = data.databricks_spark_version.latest_lts.id
  node_type_id            = data.databricks_node_type.smallest.id
  autotermination_minutes = 20
  autoscale {
    min_workers = 1
    max_workers = 50
  }

  # 批量生成PyPI类型的library块
  dynamic "library" {
    for_each = var.python_packages
    content {
      pypi {
        package = library.value
        repo    = var.pypi_repo != "" ? var.pypi_repo : null
      }
    }
  }

  # 批量生成JAR类型的library块
  dynamic "library" {
    for_each = var.jar_packages
    content {
      jar = library.value
    }
  }
}

进阶实现（支持单包自定义源） #

如果部分包需要单独指定源，可以把包列表改为对象类型，支持可选的源配置：

variable "python_packages" {
  type = list(object({
    name = string
    repo = optional(string, "")
  }))
  default = [
    {name = "conda", repo = ""},
    {name = "pymongo", repo = ""},
    {name = "kedro", repo = "artifactory-url"},
    {name = "dnspython", repo = ""}
  ]
}

resource "databricks_cluster" "shared_autoscaling" {
  # 省略集群基础配置
  dynamic "library" {
    for_each = var.python_packages
    content {
      pypi {
        package = library.value.name
        repo    = library.value.repo != "" ? library.value.repo : null
      }
    }
  }
}

内容的提问来源于stack exchange，提问作者Adesh