GitLab Runner卡在Initializing executor providers无法正常启动求助

问题描述 #

使用Docker executor时，GitLab Runner启动后卡在**"Initializing executor providers"**阶段，无法完成完整初始化。先后测试了15.11.0和16.3.0版本，均出现该问题。systemctl显示服务处于active (running)状态，但Runner无法正常执行构建任务。

版本及状态信息 #

15.11.0版本状态 #

evoro@lin-ev:~$ sudo systemctl status gitlab-runner.service 
● gitlab-runner.service - GitLab Runner
     Loaded: loaded (/etc/systemd/system/gitlab-runner.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-05-18 18:18:51 +08; 9s ago
   Main PID: 60537 (gitlab-runner)
      Tasks: 10 (limit: 18943)
     Memory: 11.3M
        CPU: 54ms
     CGroup: /system.slice/gitlab-runner.service
             └─60537 /usr/bin/gitlab-runner run --working-directory /var/lib/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --user gitlab-runner

мая 18 18:18:51 lin-ev systemd[1]: Started GitLab Runner.
мая 18 18:18:51 lin-ev gitlab-runner[60537]: Runtime platform                                    arch=amd64 os=linux pid=60537 revision=436955cb version=15.11.0
мая 18 18:18:51 lin-ev gitlab-runner[60537]: Starting multi-runner from /etc/gitlab-runner/config.toml...  builds=0
мая 18 18:18:51 lin-ev gitlab-runner[60537]: Running in system-mode.                           
мая 18 18:18:51 lin-ev gitlab-runner[60537]:                                                   
мая 18 18:18:51 lin-ev gitlab-runner[60537]: Configuration loaded                                builds=0
мая 18 18:18:51 lin-ev gitlab-runner[60537]: listen_address not defined, metrics & debug endpoints disabled  builds=0
мая 18 18:18:51 lin-ev gitlab-runner[60537]: [session_server].listen_address not defined, session endpoints disabled  builds=0
мая 18 18:18:51 lin-ev gitlab-runner[60537]: Initializing executor providers                     builds=0
evoro@lin-ev:~$ sudo gitlab-runner -v
Version:      15.11.0
Git revision: 436955cb
Git branch:   15-11-stable
GO version:   go1.19.6
Built:        2023-04-21T20:32:43+0000
OS/Arch:      linux/amd64

16.3.0版本状态 #

evoro@lin-ev:~$ sudo gitlab-runner -v
Version:      16.3.0
Git revision: 8ec04662
Git branch:   16-3-stable
GO version:   go1.20.5
Built:        2023-08-20T17:57:47+0000
OS/Arch:      linux/amd64

evoro@lin-ev:~$ sudo apt show gitlab-runner 
Package: gitlab-runner
Version: 16.3.0
Priority: optional
Section: admin
Maintainer: GitLab Inc. <support@gitlab.com>
Installed-Size: 523 MB
Provides: gitlab-ci-multi-runner
Depends: git, curl, tar, ca-certificates
Suggests: docker-engine
Conflicts: gitlab-runner-beta, gitlab-ci-multi-runner, gitlab-ci-multi-runner-beta
Replaces: gitlab-ci-multi-runner
Homepage: https://gitlab.com/gitlab-org/gitlab-runner
Download-Size: 480 MB
APT-Manual-Installed: yes
APT-Sources: https://packages.gitlab.com/runner/gitlab-runner/ubuntu jammy/main amd64 Packages
Description: GitLab Runner

evoro@lin-ev:~$ sudo systemctl status gitlab-runner.service 
● gitlab-runner.service - GitLab Runner
     Loaded: loaded (/etc/systemd/system/gitlab-runner.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-08-31 13:36:57 +08; 4min 23s ago
   Main PID: 32258 (gitlab-runner)
      Tasks: 11 (limit: 18927)
     Memory: 12.5M
        CPU: 87ms
     CGroup: /system.slice/gitlab-runner.service
             └─32258 /usr/bin/gitlab-runner run --working-directory /var/lib/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --user gitlab-runner

авг 31 13:36:57 lin-ev systemd[1]: Started GitLab Runner.
авг 31 13:36:57 lin-ev gitlab-runner[32258]: Runtime platform                                    arch=amd64 os=linux pid=32258 revision=8ec04662 version=16.3.0
авг 31 13:36:57 lin-ev gitlab-runner[32258]: Starting multi-runner from /etc/gitlab-runner/config.toml...  builds=0 max_builds=0
авг 31 13:36:57 lin-ev gitlab-runner[32258]: Running in system-mode.                           
авг 31 13:36:57 lin-ev gitlab-runner[32258]:                                                   
авг 31 13:36:57 lin-ev gitlab-runner[32258]: Configuration loaded                                builds=0 max_builds=1
авг 31 13:36:57 lin-ev gitlab-runner[32258]: listen_address not defined, metrics & debug endpoints disabled  builds=0 max_builds=1
авг 31 13:36:57 lin-ev gitlab-runner[32258]: [session_server].listen_address not defined, session endpoints disabled  builds=0 max_builds=1
авг 31 13:36:57 lin-ev gitlab-runner[32258]: Initializing executor providers                     builds=0 max_builds=1

求助问题 #

1. 如何让GitLab Runner完全启动并正常工作？
2. 导致该启动异常问题的原因是什么？

1. 修复步骤 #

步骤1：检查Docker服务状态及权限 #

• 验证Docker是否正常运行：sudo systemctl status docker，未启动则执行sudo systemctl start docker并设置开机自启sudo systemctl enable docker。
• 将gitlab-runner用户加入docker组：sudo usermod -aG docker gitlab-runner，执行后重启Runner服务：sudo systemctl restart gitlab-runner。

步骤2：验证Docker Executor配置 #

打开/etc/gitlab-runner/config.toml，检查executor配置是否正确，示例正确配置：

[[runners]]
  name = "Docker Runner"
  url = "https://your-gitlab-instance.com/"
  token = "your-runner-token"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "docker:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

修正错误配置后重启Runner服务。

步骤3：查看Runner详细日志 #

执行sudo gitlab-runner run --debug手动启动Runner，查看Debug级日志，定位卡住的具体原因，比如Docker连接超时、权限不足或镜像拉取失败等。

步骤4：清理缓存及残留容器 #

• 清理Runner缓存：sudo rm -rf /var/lib/gitlab-runner/cache/*
• 清理残留Docker容器：sudo docker rm -f $(sudo docker ps -aq)
• 重启Runner服务：sudo systemctl restart gitlab-runner

步骤5：检查系统资源 #

用top、df -h查看CPU、内存、磁盘使用情况，确保资源充足，避免因资源耗尽导致初始化卡住。

2. 可能的原因 #

• Docker服务未运行或权限不足：Docker executor依赖Docker服务，若Docker未启动，或gitlab-runner用户无Docker操作权限，会导致初始化卡住。
• 配置文件错误：config.toml中Docker executor存在语法错误、参数缺失（如镜像地址错误、token无效），会导致Executor初始化失败。
• 系统资源不足：内存、CPU耗尽，导致Runner进程无法完成Executor初始化。
• Docker镜像拉取问题：配置的默认镜像无法正常拉取（网络问题、镜像不存在），会卡住初始化流程。
• 版本兼容性问题：部分GitLab Runner版本与特定Docker版本存在兼容性冲突，可尝试升级或降级Docker版本测试。

内容的提问来源于stack exchange，提问作者Evgeny Voropaev