Jenkins构建Git检出失败（Exit Code 137）排查求助

问题核心 #

执行git reset --hard && git checkout -f dev-customer-new-20221024时，文件检出到3%（1256/34341）被终止，返回exit code 137导致部署失败；偶有部署成功但存在代码不全的情况，删除index.lock无效果。

完整构建日志 #

Started by GitLab push by topsystem
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] stage
[Pipeline] { (部署)
[Pipeline] node
Running on docker in /home/jenkins/workspace/rancher-topsystem-test
[Pipeline] {
[Pipeline] isUnix
[Pipeline] sh
+ docker inspect -f . roffe/kubectl:v1.13.2
.
[Pipeline] withDockerContainer
docker seems to be running inside container 3ffc61d09ba2047cc871ec623a451a8660793d9c5f4a2bbc9bd4f395e83357e6
$ docker run -t -d -u 0:0 --entrypoint= -w /home/jenkins/workspace/rancher-topsystem-test --volumes-from 3ffc61d09ba2047cc871ec623a451a8660793d9c5f4a2bbc9bd4f395e83357e6 -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** roffe/kubectl:v1.13.2 cat
$ docker top 9cb300e8f05ddc82b72feef767d900ced41ed4d30832b25cb278d5795695267f -eo pid,comm
[Pipeline] {
[Pipeline] withCredentials
Masking supported pattern matches of $KUBERNETES_TOKEN
[Pipeline] {
[Pipeline] script
[Pipeline] {
[Pipeline] echo
current branch: dev-customer-new-20221024
[Pipeline] writeFile
[Pipeline] sh
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
         Affected argument(s) used the following variable(s): [KUBERNETES_TOKEN]
         See https://jenkins.io/redirect/groovy-string-interpolation for details.
+ mkdir -p /root/.kube
+ echo ****
+ base64 -d
[Pipeline] sh
+ kubectl get deployments '--namespace=topsystem-test' dev-customer-new-20221024
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
dev-customer-new-20221024   1/1     1            1           42m
[Pipeline] sh
+ read pod ignore
+ grep repo-fetch-job
+ kubectl get pods '--namespace=topsystem-test-extra'
+ kubectl exec '--namespace=topsystem-test-extra' repo-fetch-job-7d66854b5b-4lrpp -- sh -c 'rm -f /topsystem.git/index; rm -f /topsystem.git/index.lock;git --git-dir=/topsystem.git gc;git --git-dir=/topsystem.git prune;find / -name index.lock |xargs rm -rf; GIT_SSH_COMMAND="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" git --git-dir=/topsystem.git fetch --all -p -f'
find: ‘/proc/1/map_files’: Operation not permitted
find: ‘/proc/49/map_files’: Operation not permitted
find: ‘/proc/50/map_files’: Operation not permitted
find: ‘/proc/57/map_files’: Operation not permitted
find: ‘/proc/58/map_files’: Operation not permitted
find: ‘/proc/60/map_files’: Operation not permitted
find: ‘/proc/61/map_files’: Operation not permitted
find: ‘/proc/116/map_files’: Operation not permitted
find: ‘/proc/117/map_files’: Operation not permitted
find: ‘/proc/118/map_files’: Operation not permitted
find: ‘/proc/119/map_files’: Operation not permitted
find: ‘/proc/8560/map_files’: Operation not permitted
find: ‘/proc/8584/map_files’: Operation not permitted
find: ‘/proc/8585/map_files’: Operation not permitted
Fetching origin
Warning: Permanently added '[frp-server.topsystem-system]:10022,[10.42.10.194]:10022' (ECDSA) to the list of known hosts.
From ssh://frp-server.topsystem-system:10022/root/topsystem
   5656a6a92f..1034b21857  dev-customer-new-20221024 -> dev-customer-new-20221024
   74783fe752..6170f7b331  dev-upload-code-20221027  -> dev-upload-code-20221027
+ read pod ignore
[Pipeline] sh
+ kubectl get pods '--namespace=topsystem-test'
+ grep dev-customer-new-20221024
+ read pod ignore
+ kubectl exec '--namespace=topsystem-test' dev-customer-new-20221024-54cd786688-5n7q6 -- sh -c 'git reset --hard && git checkout -f dev-customer-new-20221024'
Checking out files:   3% (1256/34341)   
Killed
command terminated with exit code 137
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
$ docker stop --time=1 9cb300e8f05ddc82b72feef767d900ced41ed4d30832b25cb278d5795695267f
$ docker rm -f 9cb300e8f05ddc82b72feef767d900ced41ed4d30832b25cb278d5795695267f
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
ERROR: script returned exit code 137
Finished: FAILURE

排查方向 #

1. 资源不足问题（exit code 137核心指向） #

exit code 137通常是进程因内存不足被内核杀死，优先检查：

• 目标Pod的内存配额：查看Deployment配置的resources.limits.memory，若配额低于git checkout所需内存（项目有3万+文件，检出时需大量内存处理），调高配额上限
• 宿主机内存使用：检查Pod所在节点的剩余内存，若节点内存不足，调度到其他空闲节点或扩容节点资源

2. Git仓库与文件系统问题 #

• 仓库完整性：在Pod内执行git fsck --full检查仓库是否损坏，若有损坏，直接重新克隆仓库替换现有仓库
• 文件权限：确认Pod内代码目录的读写权限，确保git进程有足够权限修改文件
• 大文件排查：用git ls-files --size-sort找出仓库中的超大文件，这类文件会导致checkout时内存占用激增，建议用Git LFS管理大文件

3. 部署脚本优化 #

• 拆分命令：把git reset --hard && git checkout -f拆分为两步执行，增加日志输出，定位具体失败环节
• 调整部署方式：避免在运行中的Pod内直接执行git操作，改为在CI环境克隆仓库后，通过镜像构建或文件同步的方式部署代码，减少运行时Pod的资源消耗

4. 安全警告处理 #

日志中提到的Groovy字符串插值传递KUBERNETES_TOKEN存在风险，解决办法：
使用Jenkins sh步骤的参数化形式传递变量，替代Groovy字符串插值，示例：

sh(script: 'mkdir -p /root/.kube && echo ${KUBERNETES_TOKEN} | base64 -d', returnStdout: false)

这种方式能避免秘密信息泄露到日志或历史记录中

内容的提问来源于stack exchange，提问作者Hao Tan