AWS ECS关联服务发现后无法启动新任务问题排查

已完成排查项 #

• ECS EC2实例已完成注册、处于活跃状态，CPU与内存资源充足，ECS agent连接正常。
• ECS服务的「Events（事件）」标签页无任何记录，不存在实例注册、任务启动/停止相关日志，无报错信息，页面完全为空。
• 已注册的EC2实例配置正确，同版本AMI在其他集群中运行状态良好。
• 任务定义配置正确，故障前1天可正常运行，之后未做任何配置变更。
• 已验证服务角色包含所有相关权限策略。

通过AWS CLI执行命令 aws ecs describe-services --services my-service --cluster my-cluster 查询服务状态，返回结果显示部署滚动状态持续为 IN_PROGRESS，无任何进展。
完整配置返回结果如下（已替换真实名称与ID信息）：

{
        "serviceArn": "arn:aws:ecs:eu-central-1:my-account-id:service/my-cluster/my-service",
        "serviceName": "my-service",
        "clusterArn": "arn:aws:ecs:eu-central-1:my-account-id:cluster/my-cluster",
        "loadBalancers": [
            {
                "targetGroupArn": "arn:aws:elasticloadbalancing:eu-central-1:my-account-id:targetgroup/my-service-lb/load-balancer-id",
                "containerName": "my-service",
                "containerPort": 8065
            }
        ],
        "serviceRegistries": [
            {
                "registryArn": "arn:aws:servicediscovery:eu-central-1:my-account-id:service/srv-srv_id",
                "containerName": "my-service",
                "containerPort": 8065
            }
        ],
        "status": "ACTIVE",
        "desiredCount": 1,
        "runningCount": 0,
        "pendingCount": 0,
        "launchType": "EC2",
        "taskDefinition": "arn:aws:ecs:eu-central-1:my-account-id:task-definition/my-service:76",
        "deploymentConfiguration": {
            "deploymentCircuitBreaker": {
                "enable": false,
                "rollback": false
            },
            "maximumPercent": 200,
            "minimumHealthyPercent": 100
        },
        "deployments": [
            {
                "id": "ecs-svc/deployment_id",
                "status": "PRIMARY",
                "taskDefinition": "arn:aws:ecs:eu-central-1:my-account-id:task-definition/my-service:76",
                "desiredCount": 1,
                "pendingCount": 0,
                "runningCount": 0,
                "failedTasks": 0,
                "createdAt": "2022-06-28T09:15:08.241000+02:00",
                "updatedAt": "2022-06-28T09:15:08.241000+02:00",
                "launchType": "EC2",
                "rolloutState": "IN_PROGRESS",
                "rolloutStateReason": "ECS deployment ecs-svc/deployment_id in progress."
            }
        ],
        "roleArn": "arn:aws:iam::my-account-id:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS",
        "events": [],
        "createdAt": "2022-06-28T09:15:08.241000+02:00",
        "placementConstraints": [],
        "placementStrategy": [
            {
                "type": "spread",
                "field": "attribute:ecs.availability-zone"
            }
        ],
        "healthCheckGracePeriodSeconds": 120,
        "schedulingStrategy": "REPLICA",
        "createdBy": "arn:aws:iam::my-account-id:role/my-role",
        "enableECSManagedTags": false,
        "propagateTags": "NONE",
        "enableExecuteCommand": false
    }

ECS服务与服务发现条目通过Terraform创建，服务定义代码如下：

resource "aws_service_discovery_service" "ecs_discovery_service" {
  name = var.service_name

  dns_config {
    namespace_id = var.service_discovery_hosted_zone_id

    dns_records {
      ttl  = 10
      type = "SRV"
    }
  }

  health_check_custom_config {
    failure_threshold = 1
  }
}

resource "aws_ecs_service" "ecs_service" {
    name                               = var.service_name
    cluster                            = var.ecs_cluster_id
    task_definition                    = var.task_definition_arn
    desired_count                      = var.desired_count
    deployment_minimum_healthy_percent = 100
    deployment_maximum_percent         = 200
    health_check_grace_period_seconds  = var.health_check_grace_period_seconds
    
    target_group_arn = aws_lb_target_group.target_group.arn
    container_name   = var.service_name
    container_port   = var.service_container_port

    
    ordered_placement_strategy {
        type  = "spread"
        field = "attribute:ecs.availability-zone"
    }

    service_registries {
        registry_arn   = aws_service_discovery_service.ecs_discovery_service.arn
        container_name = var.service_name
        container_port = var.service_container_port
    }
}

上述配置此前运行正常，在基础设施无任何配置变更的前提下，执行销毁重建操作后，ECS无法启动任何新任务。

已将问题范围定位至服务发现模块：移除配置中的service_registries配置块后，任务即可正常启动。移除服务发现配置可临时规避问题，但并非合理解决方案，目前暂未明确问题根因。

已确认服务角色包含服务发现相关的所有必要权限：

"servicediscovery:DeregisterInstance",
"servicediscovery:Get*",
"servicediscovery:List*",
"servicediscovery:RegisterInstance",
"servicediscovery:UpdateInstanceCustomHealthStatus"

咨询问题 #

• 请提供可行的排查方向建议。已参考多份官方故障排查指南，所有指南均指引查看ECS服务事件日志，但当前服务事件页为空，其余可想到的排查项均已检查完毕。
• 服务发现阻塞ECS启动新任务的可能原因是什么？原本理解ECS会在容器启动且健康检查通过后，才会向服务发现注册中心添加SRV记录，但当前场景下容器完全未被调度启动。

若需要补充其他细节信息可随时告知，非常感谢任何方向的建议。

内容的提问来源于stack exchange，提问作者Eugene