私有子网EC2无法通过NAT网关访问外网，EB拉取Docker镜像失败

部署在私有子网中的Docker平台Elastic Beanstalk应用，需要子网内EC2实例访问外网拉取Docker镜像，但拉取公开镜像时失败，报错信息如下：

2023/05/26 20:49:51.618533 [INFO] Pulling valhalla ... 
Pulling valhalla ... error

ERROR: for valhalla  manifest for gisops/valhalla:latest not found: manifest unknown: manifest unknown
manifest for gisops/valhalla:latest not found: manifest unknown: manifest unknown

2023/05/26 20:49:51.618555 [WARN] failed to execute command: docker-compose pull, retrying...
2023/05/26 20:49:51.618565 [INFO] Running command /bin/sh -c docker-compose pull
2023/05/26 20:49:54.895191 [INFO] Pulling valhalla ... 
Pulling valhalla ... error

ERROR: for valhalla  manifest for gisops/valhalla:latest not found: manifest unknown: manifest unknown
manifest for gisops/valhalla:latest not found: manifest unknown: manifest unknown

2023/05/26 20:49:54.895227 [ERROR] An error occurred during execution of command [app-deploy] - [Docker Specific Build Application]. Stop running the command. Error: failed to pull docker images: Command /bin/sh -c docker-compose pull failed with error exit status 1. Stderr:Pulling valhalla ... 
Pulling valhalla ... error

ERROR: for valhalla  manifest for gisops/valhalla:latest not found: manifest unknown: manifest unknown
manifest for gisops/valhalla:latest not found: manifest unknown: manifest unknown

相关配置信息 #

Terraform VPC配置 #

module "vpc" {
  source = "terraform-aws-modules/vpc/aws"

  name = var.main_vpc_name
  cidr = var.main_vpc_prefix

  azs                 = var.availability_zones
  private_subnets     = var.private_subnets
  public_subnets      = var.public_subnets
  database_subnets    = var.database_subnets
  elasticache_subnets = var.elasticache_subnets
  redshift_subnets    = var.redshift_subnets

  enable_nat_gateway     = true
  single_nat_gateway     = false
  one_nat_gateway_per_az = true


  enable_vpn_gateway   = false
  enable_dns_support   = true
  enable_dns_hostnames = true

  create_database_subnet_group           = true
  create_database_subnet_route_table     = true
  create_database_internet_gateway_route = true
}

Elastic Beanstalk环境配置 #

resource "aws_elastic_beanstalk_environment" "eb_env" {
  ....

  setting {
    namespace = "aws:ec2:vpc"
    name      = "VPCId"
    value     = module.vpc.vpc_id
    resource  = ""
  }

  setting {
    namespace = "aws:ec2:vpc"
    name      = "Subnets"
    value     = join(",", sort(module.vpc.private_subnets))
    resource  = ""
  }

  setting {
    namespace = "aws:ec2:vpc"
    name      = "ELBSubnets"
    value     = join(",", sort(module.vpc.public_subnets))
    resource  = ""
  }

  setting {
    namespace = "aws:ec2:vpc"
    name      = "ELBScheme"
    value     = "internal"
    resource  = ""
  }


  setting {
    namespace = "aws:ec2:vpc"
    name      = "AssociatePublicIpAddress"
    value     = "false"
    resource  = ""
  }

  setting {
    namespace = "aws:autoscaling:launchconfiguration"
    name      = "SecurityGroups"
    value     = join(",", sort([module.webtraffic-sg.security_group_id]))
    resource  = ""
  }
} 

安全组配置 #

module "webtraffic-sg" {
  source  = "terraform-aws-modules/security-group/aws"
  version = "~> 4.0"

  name        = "webtraffic-sg"
  description = "Security group for EC2 instance"
  vpc_id      = module.vpc.vpc_id

  ingress_cidr_blocks = ["0.0.0.0/0"]
  ingress_rules       = ["http-80-tcp", "https-443-tcp"]
  egress_rules        = ["all-all"]

  tags = {
    Name = "${var.project_name}-sg-webtraffic"
  }

}

路由表信息 #

排查与解决方案 #

• 验证私有子网路由表：确认EB所在私有子网关联的路由表是否包含0.0.0.0/0指向NAT网关的路由条目。当前VPC配置中开启了NAT网关，但需确保私有子网的路由表不是仅配置了数据库子网的互联网路由（create_database_internet_gateway_route仅作用于数据库子网）。
• 检查NAT网关状态：确认NAT网关处于Available状态，且部署在公有子网中，同时公有子网的路由表包含0.0.0.0/0指向Internet Gateway的路由。
• 测试DNS解析：登录私有子网内的EC2实例，执行nslookup registry-1.docker.io，验证是否能正常解析Docker Hub域名，确保VPC的enable_dns_support和enable_dns_hostnames配置生效。
• 检查网络ACL：除安全组外，确认私有子网和公有子网的网络ACL允许出站的443端口（HTTPS），以及对应的入站响应流量。
• 确认镜像标签有效性：报错提示manifest unknown，需确认gisops/valhalla:latest标签是否存在。建议改用具体版本号而非latest进行测试。
• 临时测试网络连通性：可临时将EB环境的AssociatePublicIpAddress设为true，给EC2实例分配公网IP，测试是否能拉取镜像，以此排除网络之外的问题。

内容的提问来源于stack exchange，提问作者user2650277