如何在AWS ECS服务启动失败时让部署任务失败退出
解决AWS ECS部署失败时Azure流水线不告警且自动回滚的问题
1. 修正部署断路器参数格式
你之前使用的参数格式可能存在解析问题,AWS CLI对复杂配置更适配JSON字符串写法,替换成以下命令即可启用断路器且禁止自动回滚:
aws ecs update-service \ --region "myRegion" \ --force-new-deployment \ --service myAPI \ --cluster arn:aws:ecs:xxxxx:cluster/myClusterName \ --deployment-configuration '{"deploymentCircuitBreaker": {"enable": true, "rollback": false}}'
2. 部署后轮询检查任务状态(核心步骤)
aws ecs update-service是异步执行的,命令返回成功不代表服务启动成功。需要在部署后添加轮询逻辑,一旦检测到任务启动失败就让流水线以exit 1终止。
以下是可直接复用的bash脚本,可嵌入到Azure Pipelines的脚本任务中:
#!/bin/bash REGION="myRegion" SERVICE_NAME="myAPI" CLUSTER_ARN="arn:aws:ecs:xxxxx:cluster/myClusterName" MAX_RETRIES=10 RETRY_INTERVAL=30 # 获取本次部署的最新部署ID get_latest_deployment() { aws ecs describe-services --region $REGION --cluster $CLUSTER_ARN --services $SERVICE_NAME | \ jq -r '.services[0].deployments | sort_by(.createdAt) | last.id' } DEPLOYMENT_ID=$(get_latest_deployment) echo "监控部署ID: $DEPLOYMENT_ID" for ((i=0; i<MAX_RETRIES; i++)); do # 获取当前部署状态与失败任务数 DEPLOYMENT_STATUS=$(aws ecs describe-services --region $REGION --cluster $CLUSTER_ARN --services $SERVICE_NAME | \ jq -r ".services[0].deployments[] | select(.id == \"$DEPLOYMENT_ID\") | .status") FAILED_TASKS=$(aws ecs describe-services --region $REGION --cluster $CLUSTER_ARN --services $SERVICE_NAME | \ jq -r ".services[0].deployments[] | select(.id == \"$DEPLOYMENT_ID\") | .failedTasksCount") echo "当前部署状态: $DEPLOYMENT_STATUS,失败任务数: $FAILED_TASKS" if [ "$DEPLOYMENT_STATUS" = "PRIMARY" ] && [ "$FAILED_TASKS" -eq 0 ]; then echo "部署成功,新服务已正常运行" exit 0 fi if [ "$FAILED_TASKS" -gt 0 ]; then echo "部署失败,存在启动失败的任务" exit 1 fi if [ "$DEPLOYMENT_STATUS" = "FAILED" ]; then echo "部署状态标记为失败" exit 1 fi echo "等待$RETRY_INTERVAL秒后再次检查..." sleep $RETRY_INTERVAL done echo "超过最大重试次数,部署未完成" exit 1
脚本说明:
- 先获取最新部署ID,确保监控的是本次触发的部署
- 循环检查部署状态与失败任务数,最多重试10次、每次间隔30秒(可根据服务启动时长调整)
- 若部署成为主服务且无失败任务则判定成功;出现失败任务或部署状态为FAILED时,直接以
exit 1终止流水线 - 超时未完成也判定为失败
3. Azure Pipelines集成示例
在你的azure-pipelines.yml中添加脚本任务,整合部署与状态检查逻辑:
- task: Bash@3 inputs: targetType: 'inline' script: | # 执行ECS部署命令 aws ecs update-service \ --region "myRegion" \ --force-new-deployment \ --service myAPI \ --cluster arn:aws:ecs:xxxxx:cluster/myClusterName \ --deployment-configuration '{"deploymentCircuitBreaker": {"enable": true, "rollback": false}}' # 执行状态检查逻辑 REGION="myRegion" SERVICE_NAME="myAPI" CLUSTER_ARN="arn:aws:ecs:xxxxx:cluster/myClusterName" MAX_RETRIES=10 RETRY_INTERVAL=30 get_latest_deployment() { aws ecs describe-services --region $REGION --cluster $CLUSTER_ARN --services $SERVICE_NAME | \ jq -r '.services[0].deployments | sort_by(.createdAt) | last.id' } DEPLOYMENT_ID=$(get_latest_deployment) echo "监控部署ID: $DEPLOYMENT_ID" for ((i=0; i<MAX_RETRIES; i++)); do DEPLOYMENT_STATUS=$(aws ecs describe-services --region $REGION --cluster $CLUSTER_ARN --services $SERVICE_NAME | \ jq -r ".services[0].deployments[] | select(.id == \"$DEPLOYMENT_ID\") | .status") FAILED_TASKS=$(aws ecs describe-services --region $REGION --cluster $CLUSTER_ARN --services $SERVICE_NAME | \ jq -r ".services[0].deployments[] | select(.id == \"$DEPLOYMENT_ID\") | .failedTasksCount") echo "当前部署状态: $DEPLOYMENT_STATUS,失败任务数: $FAILED_TASKS" if [ "$DEPLOYMENT_STATUS" = "PRIMARY" ] && [ "$FAILED_TASKS" -eq 0 ]; then echo "部署成功,新服务已正常运行" exit 0 fi if [ "$FAILED_TASKS" -gt 0 ]; then echo "部署失败,存在启动失败的任务" exit 1 fi if [ "$DEPLOYMENT_STATUS" = "FAILED" ]; then echo "部署状态标记为失败" exit 1 fi echo "等待$RETRY_INTERVAL秒后再次检查..." sleep $RETRY_INTERVAL done echo "超过最大重试次数,部署未完成" exit 1 env: AWS_ACCESS_KEY_ID: $(AWS_ACCESS_KEY_ID) AWS_SECRET_ACCESS_KEY: $(AWS_SECRET_ACCESS_KEY)
确保流水线已配置好AWS访问凭据,拥有ECS服务的查询权限。
内容的提问来源于stack exchange,提问作者PApostol
相关产品推荐
相关产品推荐

