CodeDeploy在ASG实例执行DownloadBundle时因‘Execution Expired’失败

问题 #

我们正在配置由CodePipeline、CodeDeploy和CodeCommit组成的部署流程。已配置好CodeCommit仓库，且可成功部署至不在Auto Scaling Group（ASG）中的EC2实例。但将该实例制作成AMI并用于创建启动模板和ASG后，ASG内的实例无法下载部署包。

已确认EC2实例绑定的IAM角色也已分配给ASG的启动模板。补充说明：ASG实例位于无NAT网关的私有子网中，我们已按照AWS文档配置了CodeDeploy与VPC结合使用的相关设置，且已配置VPC终端节点（ASG实例可访问CodeDeploy服务器的443端口）。

在DownloadBundle步骤失败前，BeforeBlockTraffic、BlockTraffic、AfterBlockTraffic和ApplicationStop步骤均执行成功。

DownloadBundle步骤失败后，AWS控制台仅显示UnknownError，ASG实例上的codedeploy-agent.log日志内容如下：

2023-06-26T17:56:11 DEBUG [codedeploy-agent(3909)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandExecutor: Executing DownloadBundle command for execution CodeDeploy/us-east-2/prod/orpheus:public005/244301334150:d-G64NUIVVN
2023-06-26T17:56:11 INFO [codedeploy-agent(3909)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandExecutor: Downloading artifact bundle from bucket 'codepipeline-us-east-2-457949292945' and key 'clc-pl-2/SourceArti/yOCp19T', version '', etag 'e27418ca132ed2ca7a7c4e8fab91472a-1'
2023-06-26T17:56:12 DEBUG [codedeploy-agent(3909)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PollHostCommand:
2023-06-26T17:56:12 INFO [codedeploy-agent(3909)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.6.0-49_rpm.
2023-06-26T17:56:57 INFO [codedeploy-agent(3909)]: [Aws::CodeDeployCommand::Client 200 45.197855 0 retries] poll_host_command(host_identifier:"arn:aws:ec2:us-east-2:244301334150:instance/i-01bb942fef9c3eda3")

2023-06-26T17:56:57 DEBUG [codedeploy-agent(3909)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: PollHostCommand: Host Command = nil
2023-06-26T17:56:58 DEBUG [codedeploy-agent(3909)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PollHostCommand:
2023-06-26T17:56:58 INFO [codedeploy-agent(3909)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.6.0-49_rpm.
2023-06-26T17:57:13 DEBUG [codedeploy-agent(3909)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PutHostCommandComplete: "Code Error"
2023-06-26T17:57:13 INFO [codedeploy-agent(3909)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.6.0-49_rpm.
2023-06-26T17:57:13 INFO [codedeploy-agent(3909)]: [Aws::CodeDeployCommand::Client 200 0.048875 0 retries] put_host_command_complete(command_status:"Failed",diagnostics:{format:"JSON",payload:"{"error_code":5,"script_name":"","message":"execution expired","log":""}"},host_command_identifier:"eyJiYXRjaElkIjoiZGE4YTM4NmRlYmI3NjljOTcyMWU5Zjk0YzllYTUzMjQvcHVibGljMDAzIiwiZGVwbG95bWVudElkIjoiQ29kZURlcGxveS91cy1lYXN0LTIvcHJvZC9vcnBoZXVzOnB1YmxpYzAwNS8yNDQzMDEzMzQxNTA6ZC1HNjROVUlWVk4iLCJob3N0SWQiOiJhcm46YXdzOmVjMjp1cy1lYXN0LTI6MjQ0MzAxMzM0MTUwOmluc3RhbmNlL2ktMDFiYjk0MmZlZjljM2VkYTMiLCJjb21tYW5kSWQiOiJBcG9sbG9EZXBsb3lDb250cm9sU2VydmljZXxhcm46YXdzOmVjMjp1cy1lYXN0LTI6MjQ0MzAxMzM0MTUwOmluc3RhbmNlL2ktMDFiYjk0MmZlZjljM2VkYTN8NXwwIiwiY29tbWFuZE5hbWUiOiJEb3dubG9hZEJ1bmRsZSIsImNvbW1hbmRJbmRleCI6NSwiYXR0ZW1wdEluZGV4IjoxfQ==")

2023-06-26T17:57:13 ERROR [codedeploy-agent(3909)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Error during perform: Seahorse::Client::NetworkingError - execution expired - /usr/share/ruby/net/http.rb:937:in `initialize'

多次检查配置未发现问题，求排查思路。

排查思路 #

• 检查S3 VPC终端节点配置：日志显示要从S3桶codepipeline-us-east-2-457949292945下载部署包，私有子网无NAT网关的情况下，必须配置S3的VPC终端节点（Gateway类型）。确认终端节点是否关联了ASG所在的私有子网，路由表是否已添加S3前缀的路由指向该终端节点；同时检查终端节点的策略是否允许实例访问目标S3桶的GetObject操作。
• 验证实例IAM权限：虽然已分配IAM角色给启动模板，需确认角色的权限策略是否包含S3的GetObject权限（针对目标codepipeline桶），以及CodeDeploy的相关权限。可以在ASG实例上执行aws s3 cp s3://codepipeline-us-east-2-457949292945/clc-pl-2/SourceArti/yOCp19T /tmp/test测试是否能下载文件，直接验证权限和网络连通性。
• 检查CodeDeploy Agent版本与兼容性：当前使用的Agent版本是OFFICIAL_1.6.0-49_rpm，该版本较旧，可能存在与新AWS服务特性不兼容的问题。尝试在ASG实例上升级CodeDeploy Agent到最新稳定版，再测试部署。
• 排查DNS解析问题：私有子网内的实例需要能正确解析S3和CodeDeploy的服务域名。检查VPC的DNS设置是否开启了enableDnsHostnames和enableDnsSupport，同时确认实例的DNS服务器配置是否正常，可通过nslookup codepipeline-us-east-2-457949292945.s3.us-east-2.amazonaws.com测试域名解析是否成功。
• 检查安全组和网络ACL：确认ASG实例的安全组是否允许出站访问S3的443端口；同时检查子网的网络ACL是否放行出站的TCP 443流量，以及入站的相关响应流量。
• 验证启动模板的AMI配置：确认制作AMI的原始实例在私有子网环境下是否能正常下载S3包（之前的测试是不在ASG的实例，可能该实例在公网环境）。可以将AMI启动的实例单独放在私有子网中，不加入ASG，测试是否能完成部署，排除ASG本身的配置影响。
• 检查CodeDeploy部署组配置：确认部署组是否正确关联了ASG，以及部署组的VPC配置是否正确指向了目标私有子网和安全组，是否启用了私有子网部署的相关选项。

内容的提问来源于stack exchange，提问作者user22117779