AWS Elastic Beanstalk Worker环境随机部署失败问题求助

问题 #

配置AWS Elastic Beanstalk Worker负载均衡环境后，出现随机部署失败的情况：多数部署可正常完成，但偶尔会失败。当环境存在多实例时，EB事件日志报错：
Unsuccessful command execution on instance id(s) 'i-*******'. Aborting the operation.

下载对应实例的日志包查看eb-engine.log，发现仅该实例出现sqsd启动失败，其余实例无异常，日志详情如下：

2022/xx/xx xx:xx:xx.590911 [INFO] Executing instruction: configureSqsd
2022/xx/xx xx:xx:xx.595059 [INFO] get sqsd conf from cfn metadata and write into sqsd conf file ...
2022/xx/xx xx:xx:xx.596414 [INFO] Executing instruction: startSqsd
2022/xx/xx xx:xx:xx.596442 [INFO] Running command /bin/sh -c systemctl show -p PartOf sqsd.service
2022/xx/xx xx:xx:xx.658752 [INFO] Running command /bin/sh -c systemctl is-active sqsd.service
2022/xx/xx xx:xx:xx.697070 [INFO] Running command /bin/sh -c systemctl start sqsd.service
2022/xx/xx xx:xx:xx.566439 [INFO] Job for sqsd.service failed because the control process exited with error code. See "systemctl status sqsd.service" and "journalctl -xe" for details.

2022/xx/xx xx:xx:xx.566462 [ERROR] An error occurred during execution of command [app-deploy] - [startSqsd]. Stop running the command. Error: startProcess Failure: starting process "sqsd" failed: Command /bin/sh -c systemctl start sqsd.service failed with error exit status 1. Stderr:Job for sqsd.service failed because the control process exited with error code. See "systemctl status sqsd.service" and "journalctl -xe" for details.

排查与解决建议 #

• 查看sqsd服务详细错误信息：在失败实例上执行以下命令，定位具体失败原因：
  • systemctl status sqsd.service：获取服务当前状态、启动失败的关键提示
  • journalctl -u sqsd.service --since "<部署时间>"：过滤部署时段的sqsd日志，找到启动失败的触发点
• 检查sqsd配置文件有效性：确认CFN元数据写入的sqsd配置文件（通常路径为/etc/aws/sqsd/config）无格式错误、缺失必要参数（如SQS队列URL、IAM角色配置）
• 验证实例IAM权限：确保Worker实例的IAM角色具备访问目标SQS队列的核心权限（sqs:ReceiveMessage、sqs:DeleteMessage等），权限生效延迟可能导致随机启动失败
• 添加sqsd启动重试逻辑：通过EB扩展配置（.ebextensions/sqsd-retry.config）添加重试机制，避免单次启动失败终止部署：

  container_commands:
    01_retry_sqsd_start:
      command: |
        for i in {1..3}; do
          systemctl start sqsd.service && break
          sleep 5
        done
      ignoreErrors: false
  

• 监控实例资源状态：检查部署时段实例的CPU、内存使用率，资源过载可能导致sqsd启动超时，可通过CloudWatch查看历史监控数据
• 升级EB平台版本：若使用较旧的Worker平台版本，升级到最新稳定版，修复已知的sqsd启动随机异常问题

内容的提问来源于stack exchange，提问作者Krishna Sunuwar