使用Terraform创建Azure Log Analytics数据收集规则失败：表不存在？

问题描述 #

我通过Terraform模块部署Azure Log Analytics工作区、数据收集端点和数据收集规则，前两者部署正常，但数据收集规则创建失败。目标Log Analytics工作区配置无误，但在stream_declaration和data flows中使用的Custom-Logs_CL流引发了问题。

报错信息 #

│ Error: creating Data Collection Rule (Subscription: "subscriptionID"
│ Resource Group Name: "rg-phr-loganalytics-dev-weu"
│ Data Collection Rule Name: "pharos-cpi-rule"): datacollectionrules.DataCollectionRulesClient#Create: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="InvalidPayload" Message="Data collection rule is invalid" Details=[{"code":"InvalidOutputTable","message":"Table for output stream 'Custom-Logs_CL' is not available for destination 'workspaceID'.","target":"properties.dataFlows[0]"},{"code":"InvalidOutputTable","message":"Custom table for stream 'Custom-Logs_CL' is not available for destination 'workspaceID'.","target":"properties.dataFlows[0]"}]
│ 
│   with module.log_analytics["pharos-cpi"].azurerm_monitor_data_collection_rule.rule,
│   on modules\log_analytics\main.tf line 16, in resource "azurerm_monitor_data_collection_rule" "rule":
│   16: resource "azurerm_monitor_data_collection_rule" "rule" {
│ 
│ creating Data Collection Rule (Subscription:
│ "Subscriptionid"
│ Resource Group Name: "rg-phr-loganalytics-dev-weu"
│ Data Collection Rule Name: "pharos-cpi-rule"):
│ datacollectionrules.DataCollectionRulesClient#Create: Failure responding to
│ request: StatusCode=400 -- Original Error: autorest/azure: Service returned
│ an error. Status=400 Code="InvalidPayload" Message="Data collection rule is
│ invalid" Details=[{"code":"InvalidOutputTable","message":"Table for output
│ stream 'Custom-Logs_CL' is not available for destination
│ 'e877bd70-2fca-429f-a7d4-183fbfe7a044'.","target":"properties.dataFlows[0]"},{"code":"InvalidOutputTable","message":"Custom
│ table for stream 'Custom-Logs_CL' is not available for destination
│ 'SubscriptionID'.","target":"properties.dataFlows[0]"}]

相关Terraform代码 #

resource "azurerm_monitor_data_collection_rule" "rule" {
  name                        = var.rule_name
  resource_group_name         = var.resource_group_name
  location                    = var.location
  tags                        = merge(var.tags, {})
  data_collection_endpoint_id = azurerm_monitor_data_collection_endpoint.endpoint.id

  destinations {
    log_analytics {
      workspace_resource_id = azurerm_log_analytics_workspace.log_analytics_workspace.id
      name                  = azurerm_log_analytics_workspace.log_analytics_workspace.workspace_id
    }

  }

  stream_declaration {
    stream_name = var.stream_declarations.name
    dynamic "column" {
      for_each = var.stream_declarations.column

      content {
        name = column.value.name
        type = column.value.type
      }
    }
  }

  dynamic "data_flow" {
    for_each = var.data_flows
    content {
      destinations       = [azurerm_log_analytics_workspace.log_analytics_workspace.workspace_id]
      streams            = data_flow.value.streams
      built_in_transform = data_flow.value.built_in_transform
      output_stream      = data_flow.value.output_stream
      transform_kql      = data_flow.value.transform_kql
    }
  }
}

核心疑问 #

我推测问题是Log Analytics工作区中对应的表不存在，尝试添加Terraform表资源块，但该资源无法创建或删除表。现咨询：

1. 该问题是否因表不存在导致？
2. 若为是，如何通过Terraform部署该表？

问题解答 #

1. 问题原因确认 #

是的，报错中的InvalidOutputTable明确指出Custom-Logs_CL对应的表在目标Log Analytics工作区中不存在，这就是数据收集规则创建失败的直接原因。Azure要求数据收集规则中指定的输出流对应的表必须已存在于工作区中。

2. Terraform部署自定义表的方案 #

目前AzureRM Terraform Provider没有原生的自定义表创建资源，可通过以下两种方式实现：

方式一：使用azurerm_log_analytics_saved_search间接创建

利用该资源执行Kusto查询来创建自定义表，示例代码：

resource "azurerm_log_analytics_saved_search" "custom_log_table" {
  name                = "CreateCustomLogsTable"
  resource_group_name = var.resource_group_name
  workspace_name      = azurerm_log_analytics_workspace.log_analytics_workspace.name
  display_name        = "Create Custom-Logs_CL Table"
  query               = <<-QUERY
    .create table Custom-Logs_CL (TimeGenerated:datetime, Message:string, Level:string)
  QUERY
}

# 确保数据收集规则在表创建后部署
resource "azurerm_monitor_data_collection_rule" "rule" {
  # 原有配置保持不变
  depends_on = [azurerm_log_analytics_saved_search.custom_log_table]
}

注意：需保证Kusto查询语法正确，且工作区拥有执行表创建的权限。

方式二：使用null_resource调用Azure CLI

通过本地执行Azure CLI命令直接创建自定义表，示例代码：

resource "null_resource" "create_custom_table" {
  provisioner "local-exec" {
    command = <<-EOT
      az monitor log-analytics workspace table create \
        --resource-group ${var.resource_group_name} \
        --workspace-name ${azurerm_log_analytics_workspace.log_analytics_workspace.name} \
        --name "Custom-Logs_CL" \
        --columns TimeGenerated:datetime Message:string Level:string
    EOT
  }

  # 仅在表不存在时触发执行
  triggers = {
    table_name = "Custom-Logs_CL"
  }
}

# 依赖表创建完成后再部署规则
resource "azurerm_monitor_data_collection_rule" "rule" {
  # 原有配置保持不变
  depends_on = [null_resource.create_custom_table]
}

注意：运行Terraform的环境需已安装并认证Azure CLI，或配置好Azure凭据。

额外注意事项 #

• 自定义表命名需遵循Azure规则：以字母开头，仅包含字母、数字和下划线，且必须以_CL结尾（表示自定义日志表）。
• 表创建完成后需等待数分钟使其完全生效，再部署数据收集规则，避免因表未就绪引发错误。

内容的提问来源于stack exchange，提问作者pieter-jan goeman