使用Terraform部署AWS Cloudwatch Event Target时出现非确定性重复问题

问题根因 #

• 未给每个aws_cloudwatch_event_target资源显式指定唯一的target_id参数。AWS CloudWatch Events规则要求同一条规则下的所有目标必须使用唯一的target_id作为标识符，未显式指定时，Terraform AWS Provider会自动生成该值，当多个目标指向同一个Lambda ARN时，自动生成的ID可能出现冲突，叠加AWS API的最终一致性特性，就会出现目标重复、缺失的非确定性问题。
• 配置的链式depends_on无实际作用，反而会增加部署时序的不可控性，进一步放大最终一致性带来的异常。

修复方案 #

修改每个aws_cloudwatch_event_target资源，为其添加唯一的target_id参数，同时移除不必要的链式依赖，修改后的配置示例如下：

resource "aws_cloudwatch_event_rule" "app_trigger" {
  name                = "app_trigger_${var.TAG_ENVIRONMENT}"
  description         = "Trigger every ${var.frequency} mins"
  schedule_expression = "rate(${var.frequency} minutes)"
  is_enabled          = var.is_active
}

resource "aws_cloudwatch_event_target" "target_A" {
  rule     = aws_cloudwatch_event_rule.app_trigger.name
  arn      = aws_lambda_function.app.arn
  target_id = "target_A"
  input = <<DOC
    {  
      "testname": "monitor.target_A"
    }
DOC
}

resource "aws_cloudwatch_event_target" "target_B" {
  rule     = aws_cloudwatch_event_rule.app_trigger.name
  arn      = aws_lambda_function.app.arn
  target_id = "target_B"
  input = <<DOC
    {  
      "testname": "monitor.target_B"
    }
DOC
}

resource "aws_cloudwatch_event_target" "target_C" {
  rule     = aws_cloudwatch_event_rule.app_trigger.name
  arn      = aws_lambda_function.app.arn
  target_id = "target_C"
  input = <<DOC
    {  
      "testname": "monitor.target_C"
    }
DOC
}

resource "aws_cloudwatch_event_target" "target_D" {
  rule     = aws_cloudwatch_event_rule.app_trigger.name
  arn      = aws_lambda_function.app.arn
  target_id = "target_D"
  input = <<DOC
    {   
      "testname": "monitor.target_D"
    }
DOC
}

注：Terraform 0.12及以上版本无需用${}包裹其他资源属性的直接引用，上述配置已简化冗余的插值语法

如果后续需要新增更多测试目标，推荐使用for_each批量生成资源，避免重复代码，同时从根源上避免target_id冲突：

locals {
  # 所有测试目标名称统一在这里维护
  test_targets = ["target_A", "target_B", "target_C", "target_D"]
}

resource "aws_cloudwatch_event_target" "test_targets" {
  for_each  = toset(local.test_targets)
  rule      = aws_cloudwatch_event_rule.app_trigger.name
  arn       = aws_lambda_function.app.arn
  target_id = each.value
  # 用jsonencode生成输入参数，避免手动写JSON的格式错误
  input = jsonencode({
    testname = "monitor.${each.value}"
  })
}

验证操作 #

1. 执行terraform state rm aws_cloudwatch_event_target.target_A aws_cloudwatch_event_target.target_B aws_cloudwatch_event_target.target_C aws_cloudwatch_event_target.target_D移除旧目标资源的状态记录，避免新旧配置冲突
2. 执行terraform apply部署修改后的配置
3. 多次重复部署验证，确认不会再出现目标重复、缺失的问题

内容的提问来源于stack exchange，提问作者c_sagan