AWS CDK配置API GW-VPCLink-CloudMap-ECS服务遇健康检查故障求助

问题背景 #

原本基于ALB的架构运行正常，为降低成本改用CloudMap/ServiceConnect，目标搭建API Gateway -> VPCLink -> CloudMap -> ECS EC2服务架构。CDK配置语法合法，但ECS任务因健康检查触发熔断机制无法启动，相关CDK代码如下：

import {InstanceClass, InstanceSize, InstanceType, Port, SecurityGroup, SubnetType, Vpc} from 'aws-cdk-lib/aws-ec2';
import {Duration, Stack} from "aws-cdk-lib";
import {
    AmiHardwareType,
    AsgCapacityProvider,
    Cluster,
    ContainerImage,
    Ec2Service,
    Ec2TaskDefinition,
    EcsOptimizedImage,
    Protocol
} from "aws-cdk-lib/aws-ecs";
import {
    CfnRoute,
    HttpApi,
    HttpConnectionType,
    HttpIntegration,
    HttpIntegrationType,
    HttpMethod,
    PayloadFormatVersion,
    VpcLink
} from "aws-cdk-lib/aws-apigatewayv2";
import {AutoScalingGroup, UpdatePolicy} from "aws-cdk-lib/aws-autoscaling";
import {SampleStackProps} from "./SampleStackProps";
import {Construct} from "constructs";
import {dnsResources} from "./SampleDns";
import {SampleRole} from "./SampleRoles";
import {HttpServiceDiscoveryIntegration} from "aws-cdk-lib/aws-apigatewayv2-integrations";
import {PrivateDnsNamespace} from "aws-cdk-lib/aws-servicediscovery";

export class SamplePublicEcsStack extends Stack {
    constructor(scope: Construct, id: string, props?: SampleStackProps) {
        super(scope, id, props);

        // VPC with minimal resources
        const vpc = new Vpc(this, 'SampleVPC', {
            natGateways: 0,
            maxAzs: 2,
            subnetConfiguration: [  // <--- Crucial: Define subnet configuration
                {
                    cidrMask: 24, // Adjust as needed
                    name: 'SamplePublicSubnet',
                    subnetType: SubnetType.PUBLIC,
                }
            ],
        });

        // ECS Cluster
        const cluster = new Cluster(this, 'SampleServerCluster', {
            vpc,
            clusterName: 'Sample-server-cluster',
        });

        const ecsSecurityGroup = new SecurityGroup(this, 'EcsSecurityGroup', {
            vpc,
            description: 'Allow inbound traffic from ALB',
            allowAllOutbound: true 
        });
        
        cluster.addAsgCapacityProvider(new AsgCapacityProvider(this, "SampleAsgCapacityProvider", {
            autoScalingGroup: new AutoScalingGroup(this, 'SampleAsg', {
                vpc: vpc,
                instanceType: InstanceType.of(InstanceClass.T4G, InstanceSize.MICRO),
                minCapacity: 1,
                maxCapacity: 1,
                machineImage: EcsOptimizedImage.amazonLinux2023(AmiHardwareType.ARM),
                updatePolicy: UpdatePolicy.replacingUpdate(),
                securityGroup: ecsSecurityGroup,
            })
        }));

        const SampleServiceImage = ContainerImage.fromEcrRepository(props?.ecrRepo!, 'latest');
        // Fargate Service
        const taskDefinition = new Ec2TaskDefinition(this, 'SampleTaskDefinition', {
            // props are required so forcing it
            taskRole: SampleRole(this, {table: props?.table!})
        });

        // container network type is default for a linux configuration
        const SampleContainer = taskDefinition.addContainer('SampleContainer', {
            image: SampleServiceImage,
            memoryLimitMiB: 512,
            portMappings: [{
                name: 'Sample-port-mapping',
                containerPort: 8080,
                // this is need for HttpAlbIntegrations
                hostPort: 8080,
                protocol: Protocol.TCP
            }],
            healthCheck: {
                command: ["CMD-SHELL", "curl -f http://127.0.0.1:8080/health || exit 1"], 
                interval: Duration.seconds(60),
                timeout: Duration.seconds(10),
                retries: 5,
                startPeriod: Duration.seconds(180)
            }
        });
        // API Gateway Integration with Load Balancer
        const vpcLinkSecurityGroup = new SecurityGroup(this, 'VpcLinkSG', {
            vpc,
            description: 'Security group for API Gateway VPC Link',
        });

        const vpcLink = new VpcLink(this, 'SampleVpcLink', {
            vpc,
            securityGroups: [vpcLinkSecurityGroup],
            subnets: vpc.selectSubnets({ subnetType: SubnetType.PUBLIC })
        });

        const namespace = new PrivateDnsNamespace(this, "SampleNameSpace", {
                name: "Sample.local",
                vpc: vpc
            }
        )

        const ec2Service = new Ec2Service(this, 'SampleService', {
            serviceName: 'Sample-service',
            cluster,
            taskDefinition,
            assignPublicIp: false,
            circuitBreaker: {enable: true, rollback: true},
            enableExecuteCommand: true, // Uncomment this for interactive debugging using session-manager-plugin
            cloudMapOptions: {
                name: 'Sample-service',
                container: SampleContainer,
                containerPort: SampleContainer.containerPort,
                cloudMapNamespace: namespace,
            },
        });

        // see documentation https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-update-security-groups.html#security-group-recommended-rules
        // troubleshooting video https://aws.amazon.com/awstv/watch/c326d7c3e9c/#chapter5
        ecsSecurityGroup.addIngressRule(
            vpcLinkSecurityGroup,
            // should be the container port
            Port.tcp(8080),
            'Allow ALB to talk to ECS'
        );

        // API Gateway
        const serviceDiscoveryInt = new HttpServiceDiscoveryIntegration('HttpServiceDiscovery', ec2Service.cloudMapService!, {
            vpcLink,
            method: HttpMethod.ANY
        });
        const httpApi = new HttpApi(this, 'HttpApi', {
            defaultIntegration: serviceDiscoveryInt,
            apiName: 'SampleApi',
            createDefaultStage: true
        });
        
        httpApi.addRoutes({
            path: '/{proxy+}',
            methods: [HttpMethod.ANY],
            integration: serviceDiscoveryInt,
        });

        dnsResources(this, httpApi);
    }
}

排查方向 #

• 健康检查命令有效性
  • Amazon Linux 2023默认未预装curl，可将健康检查命令替换为wget -qO- http://127.0.0.1:8080/health > /dev/null || exit 1，或在Dockerfile中添加RUN yum install -y curl确保工具存在。
  • 用ecs execute-command进入容器，手动执行curl http://127.0.0.1:8080/health，确认返回200状态码，检查路径是否配置正确。
• 网络与端口冲突
  • 当前配置hostPort:8080，若ECS实例上其他进程占用该端口会导致容器启动失败。建议删除hostPort配置，让ECS自动分配随机端口——CloudMap会自动注册容器的动态端口，不影响服务发现。
  • 公网子网的ECS任务设置assignPublicIp: false，若无VPC端点，实例无法拉取ECR私有镜像。需改为assignPublicIp: true，或配置ECR的VPC端点。
• 资源与权限问题
  • 检查ECS实例内存使用率，T4G.micro实例仅1GiB内存，若系统进程占用过高会导致容器无法正常启动，可在ECS控制台查看实例资源指标。
  • 确认任务角色（SampleRole）拥有ECR镜像拉取权限，以及实例能正常访问ECR仓库。
• 熔断与启动时间
  • 临时关闭熔断机制（circuitBreaker: {enable: false}），排查任务启动失败的具体原因，确认后再恢复配置。
  • 若容器启动时间较长，可适当延长健康检查的startPeriod值（比如调整为300秒）。
• CloudMap配置验证
  • 在CloudMap控制台检查私有命名空间与服务实例状态，确认ECS任务启动后能正常注册到CloudMap。

内容的提问来源于stack exchange，提问作者RobDog159