Terraform部署Fluent Bit日志路由报错排查与修正求助

需要通过Terraform部署Fluent Bit，实现将K8s特定命名空间的日志分别路由至对应Log Analytics工作区，日志存储于以命名空间命名的表中。现有两个变量映射：

• fluentbit_output_mappings：键为K8s命名空间、值为Log Analytics工作区名称的键值对
• log_analytics_workspaces：包含工作区资源组、名称等信息的对象映射（不可修改）

预期流程：

1. 通过映射关系获取对应工作区的资源组名称
2. 使用azurerm_log_analytics_workspace数据源获取工作区ID和主共享密钥
3. 渲染Fluent Bit的Output配置模板并输出可用配置

但代码运行时触发语法错误，报错位置在data "log_analytics_workspace" "mappings"的for循环定义行，提示"Invalid block definition"。

现有代码 #

variable "log_analytics_workspaces" {
  type = map(
    object({
        "LA1" = {
        resource_group_name = "LA1_RG"
        log_analytics_name  = "LA1"
        log_analytics_sku   = "PerGB2018"
        },
        "LA2" = {
        resource_group_name = "LA2_RG"
        log_analytics_name  = "LA2"
        log_analytics_sku   = "PerGB2018"
        },
        "LA3" = {
        resource_group_name = "LA3_RG"
        log_analytics_name  = "LA3"
        log_analytics_sku   = "PerGB2018"
        },
        "LA4" = {
        resource_group_name = "LA4_RG"
        log_analytics_name  = "LA4"
        log_analytics_sku   = "PerGB2018"
        }
    })
  )
}

variable "fluentbit_output_mappings" {
  description = "Map of K8s namespaces to Log Analytics workspace names"
  type        = map(string)
  default     = {
    # Add more mappings as needed. Syntax: K8s_namespace = Log_Analytics_workspace
    "ns1" = "LA1"
    "ns2" = "LA2"
    "ns3" = "LA3"
    "ns4" = "LA1"
    "ns5" = "LA2"
  }
}

data "log_analytics_workspace" "mappings" {
  for namespace, log_analytics_name in var.fluentbit_output_mappings : namespace => {
    namespace = namespace
    log_analytics_workspace_name = log_analytics_name == var.log_analytics_workspaces[log_analytics_name].log_analytics_name ? log_analytics_name : "falseValue"
    log_analytics_resource_group_name = var.log_analytics_workspaces[log_analytics_name].resource_group_name

    # Assert condition to stop processing if log_analytics_workspace_name is "falseValue"
    assert(log_analytics_workspace_name != "falseValue", "Error: log_analytics_workspace_name is invalud. Check terragrunt.hcl for correct mapping in fluentbit_output_mappings")
  }

  depends_on = [
    var.fluentbit_output_mappings,
    var.log_analytics_workspaces,
  ]
}

data "azurerm_log_analytics_workspace" "log_analytics" {
  for_each = { for namespace, log_analytics_name in var.fluentbit_output_mappings : namespace => log_analytics_name }
  
  name                = data.log_analytics_workspace.mappings[var.fluentbit_output_mappings[count.index]].log_analytics_workspace_name
  resource_group_name = data.log_analytics_workspace.mappings[var.fluentbit_output_mappings[count.index]].log_analytics_resource_group_name

  depends_on = [
    var.fluentbit_output_mappings,
    data.log_analytics_workspace.mappings,
  ]
}

data "template_file" "example" {
  count = length(var.workspace_mappings)

  template = <<YAML
[OUTPUT]
    Name azure
    Match kube.*${var.fluentbit_output_mappings.key}*
    Customer_ID ${data.azurerm_log_analytics_workspace.log_analytics[data.log_analytics_workspace.mappings[var.fluentbit_output_mappings[count.index]]].workspace_id}
    Shared_Key ${data.azurerm_log_analytics_workspace.log_analytics[data.log_analytics_workspace.mappings[var.fluentbit_output_mappings[count.index]]].primary_shared_key}
    Log_Type ${var.fluentbit_output_mappings.key}
  YAML

  depends_on = [
    var.fluentbit_output_mappings,
    data.azurerm_log_analytics_workspace.workspace,
  ]
}

output "rendered" {
  value = join("\n", data.template_file.example[*].rendered)
  sensitive = true
}

报错信息 #

╷
│ Error: Invalid block definition
│
│   on fluentbit.tf line 42, in data "log_analytics_workspace" "mappings":
│   42:   for namespace, log_analytics_name in var.fluentbit_output_mappings : namespace => {
│
│ Either a quoted string block label or an opening brace ("{") is expected
│ here.
╵
time=2024-02-09T15:49:47Z level=error msg=1 error occurred:
        * exit status 1

核心错误点 #

1. log_analytics_workspaces变量类型定义错误：原代码将类型定义为map(object({...}))但object内部嵌套多组键值对，正确结构应为map(object({resource_group_name=string, log_analytics_name=string, log_analytics_sku=string}))，每个工作区作为独立的object值。
2. 不存在的数据源log_analytics_workspace：Terraform AzureRM provider无此数据源，应使用azurerm_log_analytics_workspace，映射逻辑改用本地值(locals)处理。
3. for_each语法错误：数据源的for_each使用了错误的块语法，需直接引用合法的映射关系。
4. count与for_each混用：同时使用两种遍历方式导致引用混乱，需统一用for_each遍历映射。
5. 模板变量引用错误：模板中变量引用方式错误，应通过for_each的each.key和each.value获取对应信息。

修正后的完整代码 #

# 修正变量类型：每个工作区是独立的object，map的键为工作区名称
variable "log_analytics_workspaces" {
  type = map(object({
    resource_group_name = string
    log_analytics_name  = string
    log_analytics_sku   = string
  }))
  default = {
    "LA1" = {
      resource_group_name = "LA1_RG"
      log_analytics_name  = "LA1"
      log_analytics_sku   = "PerGB2018"
    },
    "LA2" = {
      resource_group_name = "LA2_RG"
      log_analytics_name  = "LA2"
      log_analytics_sku   = "PerGB2018"
    },
    "LA3" = {
      resource_group_name = "LA3_RG"
      log_analytics_name  = "LA3"
      log_analytics_sku   = "PerGB2018"
    },
    "LA4" = {
      resource_group_name = "LA4_RG"
      log_analytics_name  = "LA4"
      log_analytics_sku   = "PerGB2018"
    }
  }
}

variable "fluentbit_output_mappings" {
  description = "Map of K8s namespaces to Log Analytics workspace names"
  type        = map(string)
  default     = {
    "ns1" = "LA1"
    "ns2" = "LA2"
    "ns3" = "LA3"
    "ns4" = "LA1"
    "ns5" = "LA2"
  }
}

# 使用本地值处理映射关系与校验
locals {
  # 生成命名空间到工作区详情的映射，过滤不存在的工作区
  namespace_to_workspace = {
    for ns, la_name in var.fluentbit_output_mappings : ns => var.log_analytics_workspaces[la_name]
    if contains(keys(var.log_analytics_workspaces), la_name)
  }

  # 收集不存在的工作区名称
  missing_workspaces = [
    for ns, la_name in var.fluentbit_output_mappings : la_name
    if !contains(keys(var.log_analytics_workspaces), la_name)
  ]
}

# 校验：若存在不存在的工作区，抛出错误
resource "null_resource" "validate_workspaces" {
  count = length(local.missing_workspaces) > 0 ? 1 : 0

  provisioner "local-exec" {
    command = <<EOT
      echo "Error: 以下Log Analytics工作区不存在: ${join(", ", local.missing_workspaces)}"
      exit 1
    EOT
  }
}

# 获取每个工作区的ID和密钥，按工作区去重避免重复查询
data "azurerm_log_analytics_workspace" "log_analytics" {
  for_each = var.log_analytics_workspaces

  name                = each.value.log_analytics_name
  resource_group_name = each.value.resource_group_name

  depends_on = [null_resource.validate_workspaces]
}

# 渲染每个命名空间对应的Fluent Bit输出配置
data "template_file" "fluentbit_outputs" {
  for_each = local.namespace_to_workspace

  template = <<CONFIG
[OUTPUT]
    Name azure
    Match kube.*${each.key}*
    Customer_ID ${data.azurerm_log_analytics_workspace.log_analytics[each.value.log_analytics_name].workspace_id}
    Shared_Key ${data.azurerm_log_analytics_workspace.log_analytics[each.value.log_analytics_name].primary_shared_key}
    Log_Type ${each.key}
CONFIG
}

# 输出合并后的完整配置
output "rendered_fluentbit_config" {
  value = join("\n\n", [for k, v in data.template_file.fluentbit_outputs : v.rendered])
  sensitive = true
}

关键修正说明 #

1. 变量类型规范：log_analytics_workspaces改为符合Terraform规范的map(object(...))结构，每个工作区作为独立的键值对存储。
2. 本地值处理映射与校验：通过locals生成合法的命名空间-工作区映射，同时收集无效工作区名称；借助null_resource抛出明确的错误提示。
3. 数据源去重查询：直接遍历log_analytics_workspaces获取工作区信息，避免重复查询同一工作区（如ns4和ns1均指向LA1，仅查询一次）。
4. 模板渲染优化：用for_each遍历命名空间映射，模板中通过each.key获取命名空间、each.value获取工作区详情，引用数据源时直接通过工作区名称索引。
5. 输出合并：将所有模板渲染结果用换行符合并，生成完整的Fluent Bit Output配置。

内容的提问来源于stack exchange，提问作者Anton Todorov