Mac下Docker AWS ECS集成执行docker compose up时遇NoCredentialProviders错误

问题 #

尝试通过Docker ECS集成执行docker compose up部署基础设施到AWS时，触发错误：

NoCredentialProviders: no valid providers in the chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseError

已通过现有AWS配置文件创建ECS上下文：docker context create ecs dagster-ecs，但执行部署命令docker --context dagster-ecs compose --project-name dagster up时仍报错。

对应的Docker Compose文件如下：

---
version: "3.8"

services:
  # 运行dagit的服务，无业务代码，从user_code服务的gRPC加载任务
  # 由于使用QueuedRunCoordinator，dagit提交的任务会进入队列，由dagster-daemon服务执行
  dagit:
    platform: linux/amd64
    build:
      context: .
      dockerfile: ./Dockerfile
      target: dagit
    image: "$REGISTRY_URL/deploy_ecs/dagit"
    container_name: dagit
    command: "dagit -h 0.0.0.0 -p 3000 -w workspace.yaml"
    ports:
      - "3000:3000"
    environment:
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      DAGSTER_POSTGRES_DB: "postgres_db"
      DAGSTER_POSTGRES_HOSTNAME: "postgresql"
      DAGSTER_POSTGRES_PASSWORD: "postgres_password"
      DAGSTER_POSTGRES_USER: "postgres_user"
    depends_on:
      - postgresql
      - user_code
    x-aws-role:
      Statement:
        - Effect: "Allow"
          Action:
            - "ecs:DescribeTasks"
            - "ecs:StopTask"
          Resource:
            - "*"
        - Effect: "Allow"
          Action:
            - "iam:PassRole"
          Resource:
            - "*"
          Condition:
            StringLike:
              iam:PassedToService: "ecs-tasks.amazonaws.com"

  # 运行dagster-daemon进程，负责从队列中取出任务并启动，同时处理调度和传感器触发的任务
  daemon:
    platform: linux/amd64
    build:
      context: .
      dockerfile: ./Dockerfile
      target: dagster
    image: "$REGISTRY_URL/deploy_ecs/daemon"
    container_name: daemon
    command: "dagster-daemon run"
    environment:
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      DAGSTER_POSTGRES_HOSTNAME: "postgresql"
      DAGSTER_POSTGRES_USER: "postgres_user"
      DAGSTER_POSTGRES_PASSWORD: "postgres_password"
      DAGSTER_POSTGRES_DB: "postgres_db"
    depends_on:
      - postgresql
      - user_code
    x-aws-role:
      Statement:
        - Effect: "Allow"
          Action:
            - "ec2:DescribeNetworkInterfaces"
            - "ecs:DescribeTaskDefinition"
            - "ecs:DescribeTasks"
            - "ecs:ListAccountSettings"
            - "ecs:RegisterTaskDefinition"
            - "ecs:RunTask"
            - "ecs:TagResource"
            - "secretsmanager:DescribeSecret"
            - "secretsmanager:ListSecrets"
            - "secretsmanager:GetSecretValue"
          Resource:
            - "*"
        - Effect: "Allow"
          Action:
            - "iam:PassRole"
          Resource:
            - "*"
          Condition:
            StringLike:
              iam:PassedToService: "ecs-tasks.amazonaws.com"

  # 运行gRPC服务器，提供仓库信息。设置DAGSTER_CURRENT_IMAGE为自身镜像，让任务启动器用该镜像创建新容器
  # 可部署多个此类服务，每个需使用独立端口并在workspace.yaml中配置
  user_code:
    platform: linux/amd64
    build:
      context: .
      dockerfile: ./Dockerfile
      target: user_code
    image: "$REGISTRY_URL/deploy_ecs/user_code"
    container_name: user_code
    command: "dagster api grpc -h 0.0.0.0 -p 4000 -f repo.py"
    environment:
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      DAGSTER_POSTGRES_DB: "postgres_db"
      DAGSTER_POSTGRES_HOSTNAME: "postgresql"
      DAGSTER_POSTGRES_PASSWORD: "postgres_password"
      DAGSTER_POSTGRES_USER: "postgres_user"
      DAGSTER_CURRENT_IMAGE: "$REGISTRY_URL/deploy_ecs/user_code"

  # 运行PostgreSQL数据库，用于Dagster的任务存储、调度存储和事件日志存储。生产环境建议替换为RDS实例
  postgresql:
    image: postgres:11
    container_name: postgresql
    environment:
      POSTGRES_DB: "postgres_db"
      POSTGRES_PASSWORD: "postgres_password"
      POSTGRES_USER: "postgres_user"

解决方案 #

1. 检查本地AWS凭证有效性 #

• 确认~/.aws/credentials文件中存在对应配置文件的有效密钥对，格式如下：

  [your-profile-name]
  aws_access_key_id = YOUR_ACCESS_KEY
  aws_secret_access_key = YOUR_SECRET_KEY
  

• 执行aws configure list --profile your-profile-name验证凭证是否能正常读取，确保输出中access_key和secret_key不为空。

2. 确保ECS上下文使用正确的AWS配置文件 #

• 创建ECS上下文时明确指定配置文件：

  docker context create ecs dagster-ecs --profile your-profile-name
  

• 执行docker context inspect dagster-ecs查看上下文配置，确认credentials字段指向正确的配置文件。

3. 验证环境变量是否正确加载 #

• 部署前在终端导出所需环境变量：

  export AWS_ACCESS_KEY_ID=your-access-key
  export AWS_SECRET_ACCESS_KEY=your-secret-key
  export REGISTRY_URL=your-ecr-repo-url
  

• 执行echo $AWS_ACCESS_KEY_ID确认变量已正确设置，避免Compose文件中环境变量未被替换。

4. 检查IAM用户权限 #

• 确保用于部署的AWS用户拥有以下核心权限：
  • ecs:CreateCluster
  • ecs:RegisterTaskDefinition
  • ecs:RunTask
  • iam:PassRole（针对ECS任务执行角色）
  • ecr:GetAuthorizationToken（使用ECR镜像仓库时）
• 确保权限范围覆盖部署所需的所有操作，同时避免过度授权。

5. 启用凭证链verbose日志排查 #

• 设置环境变量开启详细日志，定位具体失败的凭证提供者：

  export AWS_CREDENTIALS_CHAIN_VERBOSE_ERROR=true
  

• 重新执行部署命令，根据日志信息针对性修复凭证问题。

内容的提问来源于stack exchange，提问作者user3843520