Terraform部署AWS ECS任务启动失败：JSON字符异常报错

问题情况 #

用Terraform部署AWS ECS所有资源都成功，但任务启动失败。已经验证过任务定义的JSON格式没问题。

错误信息 #

Task stopped at: 2023-08-17T18:34:10.304Z
ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve secret from asm: service call has been retried 1 time(s): invalid character '}' looking for beginning of object key string

当前任务定义 #

{
    "taskDefinitionArn": "arn:aws:ecs:ca-central-1:123:task-definition/abc-task-def:6",
    "containerDefinitions": [
        {
            "name": "abc-container",
            "image": "123.dkr.ecr.ca-central-1.amazonaws.com/gam-abc-kinesis-reader:latest",
            "cpu": 10,
            "memory": 1024,
            "portMappings": [
                {
                    "containerPort": 8080,
                    "hostPort": 8080,
                    "protocol": "tcp"
                }
            ],
            "essential": true,
            "environment": [
                {
                    "name": "ENV",
                    "value": "dev"
                }
            ],
            "mountPoints": [],
            "volumesFrom": [],
            "secrets": [
                {
                    "name": "S3_BUCKET_NAME",
                    "valueFrom": "arn:aws:secretsmanager:ca-central-1:123:secret:abc/S3_BUCKET_NAME-PgaYtE:S3_BUCKET_NAME::"
                },
                {
                    "name": "S3_SECRET_KEY",
                    "valueFrom": "arn:aws:secretsmanager:ca-central-1:123:secret:abc/S3_SECRET_KEY-XPedks:S3_SECRET_KEY::"
                },
                {
                    "name": "S3_ACCESS_KEY",
                    "valueFrom": "arn:aws:secretsmanager:ca-central-1:123:secret:abc/S3_ACCESS_KEY-j2JuEy:S3_ACCESS_KEY::"
                },
                {
                    "name": "KINESIS_ACCESS_KEY",
                    "valueFrom": "arn:aws:secretsmanager:ca-central-1:123:secret:abc/KINESIS_ACCESS_KEY-xkAnDG:KINESIS_ACCESS_KEY::"
                },
                {
                    "name": "KINESIS_SECRET_KEY",
                    "valueFrom": "arn:aws:secretsmanager:ca-central-1:123:secret:abc/KINESIS_SECRET_KEY-4arK3C:KINESIS_SECRET_KEY::"
                }
            ],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "abc-logs",
                    "awslogs-region": "ca-central-1",
                    "awslogs-stream-prefix": "abc"
                }
            }
        }
    ],
    "family": "abc-task-def",
    "executionRoleArn": "arn:aws:iam::123:role/abc-ecs_task_exec_role",
    "networkMode": "awsvpc",
    "revision": 6,
    "volumes": [],
    "status": "ACTIVE",
    "requiresAttributes": [
        {
            "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
        },
        {
            "name": "ecs.capability.execution-role-awslogs"
        },
        {
            "name": "com.amazonaws.ecs.capability.ecr-auth"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
        },
        {
            "name": "ecs.capability.secrets.asm.environment-variables"
        },
        {
            "name": "ecs.capability.execution-role-ecr-pull"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
        },
        {
            "name": "ecs.capability.task-eni"
        }
    ],
    "placementConstraints": [],
    "compatibilities": [
        "EC2",
        "FARGATE"
    ],
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "256",
    "memory": "1024",
    "registeredAt": "2023-08-17T18:32:55.815Z",
    "registeredBy": "arn:aws:sts::123:assumed-role/AWSReservedSSO_PowerAdmin_01c43d6b605087cf/ms@test.com",
    "tags": [
        {
            "key": "Project",
            "value": "abc"
        },
        {
            "key": "Owner",
            "value": "ms@test.com"
        }
    ]
}

问题原因及修复 #

错误提示里的invalid character '}' looking for beginning of object key string，说明ECS解析Secrets Manager的引用时格式出错了。

看你任务定义里的secrets部分，每个valueFrom的末尾都多了一个额外的冒号：
比如arn:aws:secretsmanager:ca-central-1:123:secret:abc/S3_BUCKET_NAME-PgaYtE:S3_BUCKET_NAME::，正确的格式应该是：

• 如果是引用JSON格式secret里的特定键：arn:aws:secretsmanager:区域:账号:secret:secret名称-随机后缀:键名:
• 如果secret本身是单个字符串值，不需要指定键名，直接用secret的ARN即可：arn:aws:secretsmanager:ca-central-1:123:secret:abc/S3_BUCKET_NAME-PgaYtE

把所有valueFrom末尾的额外冒号去掉，修正后的示例：

"secrets": [
    {
        "name": "S3_BUCKET_NAME",
        "valueFrom": "arn:aws:secretsmanager:ca-central-1:123:secret:abc/S3_BUCKET_NAME-PgaYtE:S3_BUCKET_NAME:"
    },
    {
        "name": "S3_SECRET_KEY",
        "valueFrom": "arn:aws:secretsmanager:ca-central-1:123:secret:abc/S3_SECRET_KEY-XPedks:S3_SECRET_KEY:"
    }
]

另外还要确认：

1. Secrets Manager中的secret确实是JSON格式，且包含对应的键名（比如S3_BUCKET_NAME），如果是纯文本，直接用ARN不要加键名部分。
2. 检查ecs_task_exec_role的权限，确保它有secretsmanager:GetSecretValue权限，并且资源范围包含你引用的这些secret。
3. 确认secret的ARN正确，特别是自动生成的随机后缀（比如-PgaYtE）不能省略。

内容的提问来源于stack exchange，提问作者Mohit Saxena