Jenkins Pipeline执行Docker Push时偶现unexpected EOF错误求助
Jenkins流水线首次推送ECR失败,重试成功的排查与修复
问题背景
运行在Jenkins slave节点上的流水线,负责将Docker镜像推送至AWS ECR。首次执行时会在镜像构建推送阶段失败,重试后任务可正常完成,该问题频繁出现。流水线核心代码如下:
pipeline { agent { node { label "${jenkins_node}" } } tools { maven 'maven 3.9.1' } stages { stage('GIT checkout') { steps { dir("$WORKSPACE") { checkout([$class : 'GitSCM', branches : [[name: "${BRANCH}"]], doGenerateSubmoduleConfigurations: false, extensions : [[$class : 'SubmoduleOption', disableSubmodules : false, parentCredentials : true, recursiveSubmodules: false, reference : '', timeout : 15, trackingSubmodules : false]], gitTool : 'Default', submoduleCfg : [], userRemoteConfigs : [[credentialsId: 'XXXXXXXXXX', url : 'XXXXXXXXXXXX']]]) } } } stage ('Prepare image tag name') { steps { script { echo "image tag..." IMAGE_TAG = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim() echo "DEBUG: TAG name is $IMAGE_TAG" } } } stage('Build docker image') { steps { withAWS(roleAccount:'XXXXXXXXXXXX', role:'dcp-jenkins-role') { sh ''' #!/bin/bash set -x cd $WORKSPACE mvn clean install -DskipTests ''' script{ sh """ sudo systemctl start docker sudo chmod 777 /var/run/docker.sock aws ecr get-login-password --region us-east-1 | docker login --username XXXX --password-stdin XXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com docker build -t XXXXXXXXXXXXXX:$IMAGE_TAG . docker tag XXXXXXXXXXXXXX:$IMAGE_TAG XXXXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/XXXXXXXXXXXXXX:$IMAGE_TAG docker push XXXXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/XXXXXXXXXXXXXX:$IMAGE_TAG """ } } } } } }
问题根因
- Docker服务启动异步性:
sudo systemctl start docker是异步执行命令,返回时Docker daemon可能仍在初始化,后续Docker命令(login/build/push)会因无法连接daemon而失败;重试时Docker已完全启动,因此命令正常执行。 - 权限生效延迟:
sudo chmod 777 /var/run/docker.sock的权限变更可能未即时生效,导致首次执行时Jenkins用户无权限访问Docker sock文件。 - ECR登录无重试逻辑:首次执行时AWS凭证加载或网络波动可能导致登录失败,无重试机制会直接终止流程。
修复方案
1. 等待Docker服务完全就绪
在启动Docker后添加循环检查,确保daemon正常运行后再执行后续命令:
sudo systemctl start docker # 最多等待30秒,每隔1秒检查一次Docker状态 for i in {1..30}; do if docker info >/dev/null 2>&1; then echo "Docker daemon is ready" break fi echo "Waiting for Docker daemon to start... ($i/30)" sleep 1 done # 超时未就绪则退出流程 docker info >/dev/null 2>&1 || { echo "Failed to start Docker daemon"; exit 1; }
2. 安全配置Docker权限(替代777权限)
长期来看,给sock文件设置777权限存在安全风险,建议在slave节点将Jenkins运行用户加入docker组:
sudo usermod -aG docker jenkins
执行后重启Jenkins agent服务,后续无需再修改sock权限,也能避免权限生效延迟问题。
3. 添加ECR登录重试机制
针对网络或凭证加载波动,给ECR登录命令添加重试逻辑:
# 最多重试3次登录ECR for i in {1..3}; do aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin XXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com if [ $? -eq 0 ]; then echo "ECR login succeeded" break fi echo "ECR login failed, retrying... ($i/3)" sleep 2 done # 重试失败则退出 if [ $? -ne 0 ]; then echo "Failed to login to ECR after 3 attempts" exit 1 fi
4. 拆分流水线阶段优化
将Docker环境准备逻辑单独拆分为前置阶段,确保后续构建推送阶段执行时环境已完全就绪:
stage('Prepare Docker Environment') { steps { sh ''' sudo systemctl start docker # 等待Docker就绪 for i in {1..30}; do docker info >/dev/null 2>&1 && break sleep 1 done docker info >/dev/null 2>&1 || exit 1 ''' } }
内容的提问来源于stack exchange,提问作者Gunjit Sidher
相关产品推荐
相关产品推荐

