ECS Fargate容器无法通过VPC Endpoint拉取ECR镜像求助

问题概况 #

在私有子网部署ECS Fargate容器服务，要求容器仅访问私有网络，但无法通过私有网络从私有ECR仓库拉取镜像，已卡一周。已完成以下配置：

• 创建ecr-api、ecr-dkr、网关型S3（已配置私有子网路由表）、logs的VPC端点，开启私有DNS，测试时将端点安全组开放0.0.0.0/0
• Fargate安全组开放整个VPC CIDR的入站/出站流量，同时允许与VPC端点绑定的安全组通信
• 验证IAM权限：ECR仓库策略允许所有用户操作，Fargate任务角色具备ECR、Logs等相关权限

启动容器时出现如下错误：

CannotPullContainerError: ref pull has been retried 5 time(s): failed to copy: httpReadSeeker: failed open: failed to do request: Get 956469741060.dkr.ecr.us-east-1.amazonaws.com/my-ecr-repo:latest: dial tcp 52.216.78.32:443: i/o timeout

可见容器仍尝试通过公网IP拉取镜像（VPC CIDR为10.0.0.0/16）。开放Fargate出站流量到0.0.0.0/0可正常拉取，但需避免公网访问，仅允许私有网络流量。

在私有子网启动EC2实例，对所有VPC端点执行nslookup测试均返回私有IP，说明端点配置正确，推测问题出在Fargate配置中。

现有Terraform配置 #

resource "aws_ecs_cluster" "test_sdk" {
  name = "test-sdk-${var.stage}"
}

resource "aws_ecs_task_definition" "test_task_def" {
  family                   = "test-sdk-${var.stage}"
  network_mode             = "awsvpc"
  task_role_arn            = aws_iam_role.ecs_task_execution_role.arn
  execution_role_arn       = aws_iam_role.ecs_task_execution_role.arn
  requires_compatibilities = ["FARGATE"]
  cpu                      = 4096
  memory                   = 8192
  container_definitions = jsonencode(
    [
    {
      "name": "test-container",
      "image": "${data.aws_caller_identity.self.account_id}.dkr.ecr.${var.region}.amazonaws.com/test-sdk-${var.stage}:latest",
      "essential": true,
      "portMappings": [
        {
          "containerPort": var.container_port,
          "hostPort": var.container_port
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "ecs-test-${var.stage}",
          "awslogs-region": "${var.region}",
          "awslogs-stream-prefix": "streaming"
        }
      }
    }
    ]
  )
}

resource "aws_ecs_service" "test_service" {
  name            = "test-service"
  cluster         = aws_ecs_cluster.test_sdk.id
  task_definition = aws_ecs_task_definition.test_task_def.arn
  launch_type     = "FARGATE" 
  desired_count   = 1

  network_configuration {
    subnets = [data.aws_subnet.private-1.id, data.aws_subnet.private-2.id]
    security_groups = [aws_security_group.test-sg.id]
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.test-tg.arn
    container_name   = "test-container"
    container_port   = var.container_port
  }
}

# Create a security group allowing traffic on container port
resource "aws_security_group" "test-sg" {
  name   = "test-sg-${var.stage}"
  vpc_id = data.aws_vpc.vpc.id

  ingress {
    from_port   = var.container_port
    to_port     = var.container_port
    protocol    = "tcp"
    cidr_blocks = [
       data.aws_subnet.private-1.cidr_block,
       data.aws_subnet.private-2.cidr_block
      ]
  }

  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = [
       data.aws_subnet.private-1.cidr_block,
       data.aws_subnet.private-2.cidr_block
      ] # Allow traffic from private subnet
  }

  egress {
    from_port   = var.container_port
    to_port     = var.container_port
    protocol    = "tcp"
    cidr_blocks = [
       data.aws_subnet.private-1.cidr_block,
       data.aws_subnet.private-2.cidr_block
      ] # Allow traffic from private subnet
  }
  egress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = [
       data.aws_subnet.private-1.cidr_block,
       data.aws_subnet.private-2.cidr_block
        ] # Allow traffic from private subnet
  }
}

# Create Application Load Balancer
resource "aws_lb" "test" {
  name               = "test-lb-${var.stage}"
  internal           = true
  load_balancer_type = "application"
  security_groups    = [aws_security_group.test-sg.id]
  subnets            = [data.aws_subnet.private-1.id, data.aws_subnet.private-2.id]
}

# Create Target Group
resource "aws_lb_target_group" "test-tg" {
  name     = "test-tg-${var.stage}"
  port     = var.container_port
  protocol = "HTTP"
  target_type = "ip"
  vpc_id   = data.aws_vpc.vpc.id
  health_check {
    enabled             = true
    healthy_threshold   = 2
    interval            = 90
    path                = "/"
    matcher             = "200-399"
    port                = var.container_port
    protocol            = "HTTP"
    timeout             = 40
    unhealthy_threshold = 2
  }
}

# Create listener
resource "aws_lb_listener" "test-listener" {
  load_balancer_arn = aws_lb.test.arn
  port              = var.container_port
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.test-tg.arn
  }
}

# IAM
resource "aws_iam_role" "ecs_task_execution_role" {
  name = "tf-${var.project}-${var.stage}-ecs-task-execution-role"
  assume_role_policy = data.aws_iam_policy_document.ecs_assume_role_policy.json
  inline_policy {
    name = "test-sdk-ecr-repo-policy"
    policy = jsonencode({
      "Version" : "2012-10-17",
      "Statement" : [
        {
          "Effect" : "Allow",
          "Action" : [
            "ecr:GetAuthorizationToken",
            "ecr:BatchCheckLayerAvailability",
            "ecr:GetDownloadUrlForLayer",
            "ecr:BatchGetImage",
            "logs:CreateLogGroup",
            "logs:DescribeLogGroups",
            "logs:DescribeLogStreams",
            "logs:CreateLogStream",
            "logs:PutLogEvents",
            "secretsmanager:GetSecretValue",
            "events:PutEvents"
          ],
          "Resource" : "*"
        }
      ]
    })
  }
}

data "aws_iam_policy_document" "ecs_assume_role_policy" {
  statement {
    actions = [
      "sts:AssumeRole"
    ]
    effect = "Allow"
    principals {
      type        = "Service"
      identifiers = ["ecs-tasks.amazonaws.com"]
    }
  }
}

解决方案 #

1. 修正安全组出站规则 #

现有安全组仅允许流量到指定的两个私有子网，但VPC端点的IP可能不在这些子网范围内，且未开放DNS解析所需的53端口，导致Fargate无法解析ECR域名到私有IP，只能尝试公网访问。

修改安全组配置：

• 将出站规则的CIDR范围改为整个VPC的CIDR，确保能访问所有VPC内的端点资源
• 添加DNS解析所需的UDP/TCP 53端口出站规则

修正后的安全组代码：

resource "aws_security_group" "test-sg" {
  name   = "test-sg-${var.stage}"
  vpc_id = data.aws_vpc.vpc.id

  ingress {
    from_port   = var.container_port
    to_port     = var.container_port
    protocol    = "tcp"
    cidr_blocks = [
       data.aws_subnet.private-1.cidr_block,
       data.aws_subnet.private-2.cidr_block
      ]
  }

  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = [
       data.aws_subnet.private-1.cidr_block,
       data.aws_subnet.private-2.cidr_block
      ]
  }

  # 允许到整个VPC的容器端口流量
  egress {
    from_port   = var.container_port
    to_port     = var.container_port
    protocol    = "tcp"
    cidr_blocks = [data.aws_vpc.vpc.cidr_block]
  }

  # 允许到整个VPC的443端口流量（访问ECR、Logs等VPC端点）
  egress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = [data.aws_vpc.vpc.cidr_block]
  }

  # 允许DNS解析（UDP/TCP 53端口）
  egress {
    from_port   = 53
    to_port     = 53
    protocol    = "udp"
    cidr_blocks = [data.aws_vpc.vpc.cidr_block]
  }

  egress {
    from_port   = 53
    to_port     = 53
    protocol    = "tcp"
    cidr_blocks = [data.aws_vpc.vpc.cidr_block]
  }
}

2. 确认VPC端点子网配置 #

检查ecr-api、ecr-dkr等VPC端点是否部署在Fargate任务所在的私有子网中，确保这些子网的路由表已添加指向VPC端点的路由条目。

3. 验证S3网关端点路由 #

确认私有子网的路由表已添加S3前缀列表（pl-xxxx）指向S3网关端点的路由，因为ECR镜像层存储在S3，Fargate需要通过该端点访问镜像层数据。

内容的提问来源于stack exchange，提问作者Niv Shitrit