ECS Fargate服务创建时任务部分失败后恢复稳定的问题排查

现象 #

使用ECS Fargate、ALB和实例IP型目标组部署服务时，任务出现间歇性启动失败：

• 失败任务报错如下：

Task stopped at: 2023-12-15T06:06:25.165Z
ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3 time(s): RequestError: send request failed caused by: Post "https://api.ecr.ap-south-1.amazonaws.com/": dial tcp 13.234.9.92:443: i/o timeout. Please check your task network configuration.

• 失败后系统会自动创建新任务，一段时间后任务可正常运行并进入稳定状态
• 调整期望任务数时呈现规律：设为2时1个失败，设为3时2个失败

任务定义 #

{
       "taskDefinitionArn": "arn:aws:ecs:ap-south-1:6xxxx5:task-definition/qrc-bg-test-v1-TaskDefinition:2",
       "containerDefinitions": [
           {
               "name": "qrc-bg-test-v1-Container",
               "image": "6xxx5.dkr.ecr.ap-south-1.amazonaws.com/qrc:latest",
               "cpu": 0,
               "links": [],
               "portMappings": [
                   {
                       "containerPort": 5000,
                       "hostPort": 5000,
                       "protocol": "tcp"
                   }
               ],
               "essential": true,
               "entryPoint": [],
               "command": [],
               "environment": [],
               "environmentFiles": [],
               "mountPoints": [],
               "volumesFrom": [],
               "secrets": [],
               "dnsServers": [],
               "dnsSearchDomains": [],
               "extraHosts": [],
               "dockerSecurityOptions": [],
               "dockerLabels": {},
               "ulimits": [],
               "logConfiguration": {
                   "logDriver": "awslogs",
                   "options": {
                       "awslogs-create-group": "true",
                       "awslogs-group": "/ecs/qrc-bg-test-v1TaskDefinition",
                       "awslogs-region": "ap-south-1",
                       "awslogs-stream-prefix": "ecs"
                   },
                   "secretOptions": []
               },
               "systemControls": []
           }
       ],
       "family": "qrc-bg-test-v1-TaskDefinition",
       "taskRoleArn": "arn:aws:iam::6xxxx5:role/ecsTaskExecutionRole",
       "executionRoleArn": "arn:aws:iam::6xxx5:role/ecsTaskExecutionRole",
       "networkMode": "awsvpc",
       "revision": 2,
       "volumes": [],
       "status": "ACTIVE",
       "requiresAttributes": [
           {
               "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
           },
           {
               "name": "ecs.capability.execution-role-awslogs"
           },
           {
               "name": "com.amazonaws.ecs.capability.ecr-auth"
           },
           {
               "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
           },
           {
               "name": "com.amazonaws.ecs.capability.docker-remote-api.1.17"
           },
           {
               "name": "com.amazonaws.ecs.capability.task-iam-role"
           },
           {
               "name": "ecs.capability.execution-role-ecr-pull"
           },
           {
               "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
           },
           {
               "name": "ecs.capability.task-eni"
           },
           {
               "name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"
           }
       ],
       "placementConstraints": [],
       "compatibilities": [
           "EC2",
           "FARGATE"
       ],
       "requiresCompatibilities": [
           "FARGATE"
       ],
       "cpu": "1024",
       "memory": "3072",
       "registeredAt": "2023-12-15T06:00:50.419Z",
       "registeredBy": "arn:aws:sts::6xxxxx5:assumed-role/B_Role",
       "tags": [] 
}

排查与解决思路 #

1. 网络配置核查 #

• VPC端点设置：为ECR创建对应区域的VPC端点（com.amazonaws.ap-south-1.ecr.api和com.amazonaws.ap-south-1.ecr.dkr），避免任务通过公网访问ECR时的网络波动。已配置的话，检查端点安全组是否允许任务安全组的443端口入站。
• 任务安全组：确认任务使用的安全组出站规则允许HTTPS（443端口）流量，无论访问公网还是VPC端点。
• 子网路由：检查任务所在子网的路由表：走公网需有指向互联网网关（IGW）的路由；用VPC端点需添加指向端点的路由条目。

2. IAM角色权限验证 #

• 确认ecsTaskExecutionRole已附加AmazonEC2ContainerRegistryReadOnly策略，且信任关系配置正确（允许ecs-tasks.amazonaws.com扮演该角色）。
• 验证角色权限覆盖ap-south-1区域的ECR资源。

3. 任务启动参数调整 #

• 调整服务部署配置中的任务启动超时时间，给任务更多初始化时间应对网络波动；若支持，可在任务定义中配置ECR拉取的重试策略。

4. 区域服务状态检查 #

• 查看AWS服务健康仪表盘，确认ap-south-1区域的ECR、Fargate服务是否存在临时故障。

内容的提问来源于stack exchange，提问作者vmax