如何通过Terraform在Azure Databricks中启用Unity Catalog

你的现有Terraform代码缺少Unity Catalog启用所需的核心组件，还存在几处配置错误，以下是具体问题和修复方案：

核心问题点 #

1. 未定义Unity Catalog元存储（Metastore）：元存储是Unity Catalog的核心组件，用于统一管理数据资产元数据，必须显式创建并关联存储位置。
2. 未绑定工作区与元存储：创建元存储后，需要将Databricks工作区与元存储关联，才能在工作区中启用Unity Catalog功能。
3. 资源引用错误：azurerm_role_assignment.databricks_admin中引用了不存在的工作区资源databrit_log，正确引用应为你定义的databricks_cdf_audit_log。

完整修复后的Terraform代码 #

# 保留现有资源，修正资源引用错误
resource "azurerm_databricks_workspace" "databricks_cdf_audit_log" {
  name                = "databricks-cdf-audit-log"
  resource_group_name = data.azurerm_resource_group.cdf_audit_log_rg.name
  location            = var.databricks_location
  sku                 = "premium"

  tags = {
    Environment = var.environment
    service     = var.service
    team        = var.team
  }
}

resource "azurerm_databricks_access_connector" "databricks_it_access_connector" {
  name                = "databricks-cdf-connector"
  resource_group_name = data.azurerm_resource_group.dit_log_rg.name
  location            = var.databricks_location

  identity {
    type = "SystemAssigned"
  }

  tags = {
    Environment = var.environment
    service     = var.service
    team        = var.team
  }
}

resource "azurerm_storage_account" "storage_account" {
  name                     = "${var.storage_account_prefix}auditmeta"
  resource_group_name      = data.azurerm_resource_group.cdf_audit_log_rg.name
  location                 = var.databricks_location
  account_tier             = "Standard"
  account_replication_type = "LRS"

  tags = {
    Environment = var.environment
    service     = var.service
    team        = var.team
  }
}

resource "azurerm_storage_container" "storage_contain" {
  name                  = "databrdata"
  storage_account_name  = azurerm_storage_account.storage_account.name
  container_access_type = "private"
}

# 修正资源引用错误
resource "azurerm_role_assignment" "databricks_admin" {
  scope                = azurerm_databricks_workspace.databricks_cdf_audit_log.id
  role_definition_name = "Owner"
  principal_id         = data.azuread_group.access_audit_members.object_id
}

resource "azurerm_role_assignment" "databricks_connector_role_assignment" {
  scope                = azurerm_storage_account.storage_account.id
  role_definition_name = "Storage Blob Data Contributor"
  principal_id         = azurerm_databricks_access_connector.databricks_it_access_connector.identity[0].principal_id
}

# 新增：创建Unity Catalog元存储
resource "azurerm_databricks_metastore" "uc_metastore" {
  name                = "databricks-cdf-audit-metastore"
  resource_group_name = data.azurerm_resource_group.cdf_audit_log_rg.name
  location            = var.databricks_location
  storage_root_path   = "abfss://${azurerm_storage_container.storage_contain.name}@${azurerm_storage_account.storage_account.name}.dfs.core.windows.net/uc-metadata"
  access_connector_id = azurerm_databricks_access_connector.databricks_it_access_connector.id

  tags = {
    Environment = var.environment
    service     = var.service
    team        = var.team
  }
}

# 新增：将Databricks工作区关联到元存储
resource "azurerm_databricks_metastore_assignment" "workspace_metastore_bind" {
  metastore_id         = azurerm_databricks_metastore.uc_metastore.id
  workspace_id         = azurerm_databricks_workspace.databricks_cdf_audit_log.id
  default_catalog_name = "hive_metastore" # 可自定义默认目录名称
}

# 补充：为元存储分配数据访问权限（确保访问连接器能读写存储）
resource "azurerm_databricks_metastore_data_access" "metastore_data_access" {
  metastore_id        = azurerm_databricks_metastore.uc_metastore.id
  name                = "databricks-cdf-metastore-data-access"
  access_connector_id = azurerm_databricks_access_connector.databricks_it_access_connector.id
}

关键配置说明 #

• storage_root_path：指定元存储用于存储元数据的根路径，格式为abfss://<容器名>@<存储账户名>.dfs.core.windows.net/<子路径>，Terraform会自动创建子路径。
• access_connector_id：关联访问连接器，用于元存储访问存储账户时的身份验证，避免使用密钥认证。
• metastore_assignment：完成工作区与元存储的绑定，这是在工作区中启用Unity Catalog的必要步骤。
• 权限：访问连接器需拥有存储容器的Storage Blob Data Contributor权限，确保元存储能正常读写元数据文件。

内容的提问来源于stack exchange，提问作者ZZZSharePoint