使用VPC终端节点时ECR出现CannotPullContainerError问题排查
问题描述
我用AWS CDK创建了一个ECS栈,代码如下:
import * as cdk from 'aws-cdk-lib' import { Construct } from 'constructs' import * as ec2 from 'aws-cdk-lib/aws-ec2' import * as ecs from 'aws-cdk-lib/aws-ecs' import path = require('path') export class CdkPlaygroundEcsAuroraStack extends cdk.Stack { constructor(scope: Construct, id: string, props?: cdk.StackProps) { super(scope, id, props) // VPC const vpc = new ec2.Vpc(this, 'ecs-vpc', { maxAzs: 2, subnetConfiguration: [ { name: 'public', subnetType: cdk.aws_ec2.SubnetType.PUBLIC }, { name: 'private', subnetType: cdk.aws_ec2.SubnetType.PRIVATE_ISOLATED } ] }) // VPC Endpoints const ecrVpcEndpoint = new ec2.InterfaceVpcEndpoint(this, 'ECRVpcEndpoint', { vpc, service: ec2.InterfaceVpcEndpointAwsService.ECR, privateDnsEnabled: true }) const cloudWatchVpcEndpoint = new ec2.InterfaceVpcEndpoint(this, 'cloudWatchVpcEndpoint', { vpc, service: ec2.InterfaceVpcEndpointAwsService.CLOUDWATCH, privateDnsEnabled: true }) const s3GatewayEndpoint = new ec2.GatewayVpcEndpoint(this, 'S3GatewayEndpoint', { service: ec2.GatewayVpcEndpointAwsService.S3, vpc, subnets: [{ subnetType: cdk.aws_ec2.SubnetType.PRIVATE_ISOLATED }] }) // ECS Cluster const ecsCluster = new ecs.Cluster(this, 'ecs-cluster', { vpc, enableFargateCapacityProviders: true, clusterName: 'ecs-cluster' }) const taskDefinition = new ecs.FargateTaskDefinition(this, 'task1', { cpu: 256, memoryLimitMiB: 512 }) const container = taskDefinition.addContainer('container', { image: ecs.ContainerImage.fromAsset(path.resolve(__dirname, '../server')), memoryLimitMiB: 256 }) const service = new ecs.FargateService(this, 'server', { cluster: ecsCluster, taskDefinition, serviceName: 'server', vpcSubnets: { subnetType: cdk.aws_ec2.SubnetType.PRIVATE_ISOLATED } }) } }
该栈包含一个运行在无NAT网关的私有子网中的ECS任务,我已配置ECR、S3和CloudWatch的VPC终端节点,希望通过这些终端节点拉取镜像,但容器任务无法拉取镜像,报错如下:
CannotPullContainerError: pull image manifest has been retried 5 time(s): failed to resolve ref 650289367947.dkr.ecr.eu-central-1.amazonaws.com/cdk-hnb659fds-container-assets-650289367947-eu-central-1:d71b37a3ce0b63a08136ce5b816ea2d5d4b677fc8e3ee8b6eda9738d7c16ebb1: failed to do request: Head "https://650289367947.dkr.ecr.eu-central-1.amazonaws.com/v2/cdk-hnb659fds-container-assets-650289367947-eu-central-1/manifests/d71b37a3ce0b63a08136ce5b816ea2d5d4b677fc8e3ee8b6eda9738d7c16ebb1": dial tcp 3.121.190.14:443: i/o timeout
从错误信息来看,ECS在尝试通过公网访问ECR,为何不使用VPC接口终端节点?如何修改解决该问题?
原因分析
- 缺少ECR Docker接口终端节点:当前仅创建了ECR的API接口终端节点(对应
ec2.InterfaceVpcEndpointAwsService.ECR),但拉取ECR镜像需要两类终端节点:ECR API终端节点用于镜像管理操作,ECR Docker终端节点(对应ec2.InterfaceVpcEndpointAwsService.ECR_DOCKER)才是镜像拉取时的实际访问端点。 - 终端节点子网部署范围问题:现有ECR、CloudWatch接口终端节点默认部署到所有子网类型,但需确保它们在
PRIVATE_ISOLATED子网中存在端点,否则Fargate任务所在子网无法访问。
解决方案
修改CDK代码,添加ECR Docker接口终端节点,并显式指定所有必要终端节点部署到PRIVATE_ISOLATED子网:
// 在VPC Endpoints部分添加ECR Docker终端节点 const ecrDockerVpcEndpoint = new ec2.InterfaceVpcEndpoint(this, 'ECRDockerVpcEndpoint', { vpc, service: ec2.InterfaceVpcEndpointAwsService.ECR_DOCKER, privateDnsEnabled: true, subnets: { subnetType: cdk.aws_ec2.SubnetType.PRIVATE_ISOLATED } }) // 修改现有ECR终端节点的子网配置 const ecrVpcEndpoint = new ec2.InterfaceVpcEndpoint(this, 'ECRVpcEndpoint', { vpc, service: ec2.InterfaceVpcEndpointAwsService.ECR, privateDnsEnabled: true, subnets: { subnetType: cdk.aws_ec2.SubnetType.PRIVATE_ISOLATED } }) // 修改现有CloudWatch终端节点的子网配置 const cloudWatchVpcEndpoint = new ec2.InterfaceVpcEndpoint(this, 'cloudWatchVpcEndpoint', { vpc, service: ec2.InterfaceVpcEndpointAwsService.CLOUDWATCH, privateDnsEnabled: true, subnets: { subnetType: cdk.aws_ec2.SubnetType.PRIVATE_ISOLATED } })
确认S3网关终端节点已关联到PRIVATE_ISOLATED子网的路由表(CDK创建网关终端节点时会自动关联,若存在自定义路由表需手动检查)。
最后重新部署CDK栈:
cdk deploy
修改完成后,Fargate任务即可通过VPC终端节点访问ECR的Docker端点,成功拉取镜像。
内容的提问来源于stack exchange,提问作者Tobias S.
相关产品推荐
相关产品推荐

