配置CloudWatch告警检测ECS任务数异常时提示数据不足,求排查
问题:ECS任务期望数大于运行数的CloudWatch告警提示数据不足
我用以下Terraform代码配置CloudWatch告警,期望当ECS任务的期望数量大于运行数量时触发告警:
resource "aws_cloudwatch_metric_alarm" "ecs_task_count_alarm" { for_each = toset(var.ecs_services) # 将列表转为集合以遍历唯一服务名 alarm_name = "ECS_Tasks_Running_Check_${each.key}" comparison_operator = "GreaterThanThreshold" evaluation_periods = 1 threshold = 0 alarm_description = "当ECS服务${each.key}的DesiredTaskCount大于RunningTaskCount时触发告警。" # RunningTaskCount指标查询 metric_query { id = "running_task_count" metric { metric_name = "RunningTaskCount" namespace = "AWS/ECS" period = 60 # 1分钟 stat = "Average" dimensions = { ClusterName = var.ecs_cluster_name ServiceName = each.key } } } # DesiredTaskCount指标查询 metric_query { id = "desired_task_count" metric { metric_name = "DesiredTaskCount" namespace = "AWS/ECS" period = 60 # 1分钟 stat = "Average" dimensions = { ClusterName = var.ecs_cluster_name ServiceName = each.key } } } # 计算期望与运行任务数差值的数学表达式 metric_query { id = "number_of_tasks_we_want_but_arent_getting" expression = "desired_task_count - running_task_count" label = "期望与运行任务数的差值" return_data = true } alarm_actions = [aws_sns_topic.sns_alarms.arn] insufficient_data_actions = [aws_sns_topic.sns_alarms.arn] ok_actions = [aws_sns_topic.sns_alarms.arn] }
但目前该告警提示数据不足,其中ecs_services为服务名称列表["foo", "bar"],ecs_cluster_name为运行该服务的集群名称。请问我遗漏了什么配置?
解决方法
导致数据不足的核心问题及修复方案如下:
统计指标类型错误:
RunningTaskCount和DesiredTaskCount是瞬时计数型指标,代表某一时刻的任务数量,使用Average统计会导致数据失真(比如1分钟内数值无变化时平均值有效,但如果指标波动或上报点不足,就会出现无有效数据的情况)。应该替换为Sum或Maximum,这两种统计类型更适配这类离散计数指标:stat = "Sum" # 或 "Maximum"维度大小写匹配检查:CloudWatch的维度是大小写敏感的,需确认
var.ecs_cluster_name和each.key(服务名)与ECS控制台显示的集群、服务名称完全一致,包括大小写、特殊字符,哪怕一个字符不匹配都会找不到指标数据。指标延迟问题:ECS的CloudWatch指标通常有1-5分钟的上报延迟,刚创建的告警或服务需要等待一段时间,待指标数据正常上报后再观察告警状态。
修改后的核心代码片段(仅调整stat字段):
# RunningTaskCount指标查询 metric_query { id = "running_task_count" metric { metric_name = "RunningTaskCount" namespace = "AWS/ECS" period = 60 stat = "Sum" # 替换为Sum或Maximum dimensions = { ClusterName = var.ecs_cluster_name ServiceName = each.key } } } # DesiredTaskCount指标查询 metric_query { id = "desired_task_count" metric { metric_name = "DesiredTaskCount" namespace = "AWS/ECS" period = 60 stat = "Sum" # 替换为Sum或Maximum dimensions = { ClusterName = var.ecs_cluster_name ServiceName = each.key } } }
内容的提问来源于stack exchange,提问作者Jordan Mackie
相关产品推荐
相关产品推荐

