AWS ECS WaitUntilTasksRunningWithContext返回ResourceNotReady解决方案咨询

问题场景 #

调用AWS Go SDK的WaitUntilTasksRunningWithContext等待EC2类型ECS任务启动时，频繁收到以下错误，但控制台确认任务已处于Running状态：

error waiting AWS ECS Task "arn:aws:ecs:eu-west-1:123456789012:task/ecs-cluster/22452be490a149e781a596a7847dd27c" to be in "Running" state: ResourceNotReady: failed waiting for successful resource state

执行代码：

input := ecs.DescribeTasksInput{
        Cluster: &cluster,
        Tasks:   []*string{&taskARN},
    }

    err = a.ecsSvc.WaitUntilTasksRunningWithContext(ctx, &input)
    if err != nil {
        return fmt.Errorf(`error waiting AWS Fargate Task %q to be in "Running" state: %w`, taskARN, err)
    }

推测是SDK默认waiter的轮询参数不合理，导致提前判定任务未就绪。以下是两种解决方案：

方案一：自定义WaiterOption调整轮询策略（优先选择） #

默认waiter的轮询间隔、重试次数可能无法覆盖EC2任务的启动耗时（涉及实例调度、镜像拉取等），通过WaiterOption自定义参数可以从根源解决问题：

import (
    "context"
    "fmt"
    "time"

    "github.com/aws/aws-sdk-go-v2/service/ecs"
    "github.com/aws/aws-sdk-go-v2/service/ecs/types"
    "github.com/aws/smithy-go/waiter"
)

// ...

input := ecs.DescribeTasksInput{
    Cluster: &cluster,
    Tasks:   []*string{&taskARN},
}

// 自定义waiter配置：增加重试次数+指数退避延迟
waiterOpts := []func(*waiter.Config){
    waiter.WithWaiterMaxAttempts(30), // 默认一般是20次，根据实际耗时调整
    waiter.WithWaiterDelay(func(attempt int) time.Duration {
        // 指数退避，避免频繁调用API，最大延迟10秒
        delay := time.Duration(1<<attempt) * time.Second
        if delay > 10*time.Second {
            delay = 10*time.Second
        }
        return delay
    }),
}

err = a.ecsSvc.WaitUntilTasksRunningWithContext(ctx, &input, waiterOpts...)
if err != nil {
    // 报错后主动检查任务实际状态，避免误判
    descResp, descErr := a.ecsSvc.DescribeTasksWithContext(ctx, &input)
    if descErr == nil && len(descResp.Tasks) > 0 {
        if descResp.Tasks[0].LastStatus == types.TaskStatusRunning {
            return nil // 任务实际已运行，忽略waiter错误
        }
    }
    return fmt.Errorf(`error waiting AWS ECS Task %q to be in "Running" state: %w`, taskARN, err)
}

方案二：对Waiter调用进行重试（备用） #

如果自定义参数后仍偶发问题，可以针对ResourceNotReady错误做重试逻辑：

import (
    "context"
    "errors"
    "fmt"
    "time"

    "github.com/aws/aws-sdk-go-v2/service/ecs"
    "github.com/aws/aws-sdk-go-v2/service/ecs/types"
    "github.com/aws/smithy-go"
)

// ...

input := ecs.DescribeTasksInput{
    Cluster: &cluster,
    Tasks:   []*string{&taskARN},
}

maxRetries := 3
var err error

for retryCount := 0; retryCount <= maxRetries; retryCount++ {
    err = a.ecsSvc.WaitUntilTasksRunningWithContext(ctx, &input)
    if err == nil {
        break
    }

    // 仅处理ResourceNotReady错误
    var apiErr smithy.APIError
    if errors.As(err, &apiErr) && apiErr.ErrorCode() == "ResourceNotReady" {
        if retryCount == maxRetries {
            // 最后一次重试失败，检查任务实际状态
            descResp, descErr := a.ecsSvc.DescribeTasksWithContext(ctx, &input)
            if descErr == nil && len(descResp.Tasks) > 0 && descResp.Tasks[0].LastStatus == types.TaskStatusRunning {
                return nil
            }
            return fmt.Errorf(`failed waiting for ECS Task %q after %d retries: %w`, taskARN, maxRetries, err)
        }
        // 指数退避等待后重试
        time.Sleep(time.Duration(1<<retryCount) * time.Second)
        continue
    }

    // 非目标错误，直接返回
    return fmt.Errorf(`error waiting AWS ECS Task %q to be in "Running" state: %w`, taskARN, err)
}

核心注意事项 #

1. 无论用哪种方案，报错后必须主动调用DescribeTasks验证任务状态，避免SDK waiter的误判导致程序异常
2. EC2类型任务启动耗时波动大，需根据自身业务场景调整重试次数、延迟时间
3. 不要设置过大的重试次数，避免不必要的程序阻塞

内容的提问来源于stack exchange，提问作者mieliespoor