GitLab中‘Job failed (system failure): aborted: <nil>’错误原因排查

Random failures like this are tricky, but let’s walk through the most likely causes and fixes based on my experience with GitLab Runner issues:

• Outdated GitLab Runner Version
  Your runner is on version 11.8.0, which is several years old (released in 2019). This version has known bugs and compatibility gaps with newer GitLab instances. Vague abort errors like this are often patched in later stable releases—upgrading to the latest supported version should be your first troubleshooting step.

• System Resource Constraints
  If BUILDMACHINE01 runs out of memory (OOM) or hits CPU limits mid-job, the OS might abruptly terminate the runner process, leading to that generic aborted: <nil> message. Check your system logs (like /var/log/syslog or dmesg on Linux) for OOM killer entries or sustained high load averages. You can also monitor resource usage during a job with top or htop to see if resources are maxing out.

• Shell Executor Permissions Issues
  The gitlab-runner user might lack sufficient permissions to access the build directory, execute compilation scripts, or interact with Git repositories. Run a simple test job that runs whoami and ls -la in the build folder to verify access. Also, ensure the runner has access to all required tools (compilers, package managers) without needing sudo privileges.

• Corrupted Build Directory
  Leftover files from previous jobs can cause conflicts or filesystem corruption over time. Try manually cleaning the runner’s default build directory (usually /home/gitlab-runner/builds/ on Linux) or enable the clean_builds option in your config.toml to auto-clean between jobs.

• Intermittent Network Issues
  If the runner loses connection to your GitLab server mid-job, it may abort with a vague error. Check the runner logs (typically /var/log/gitlab-runner.log) for network timeouts, connection refused messages, or SSL errors. You can also test connectivity from BUILDMACHINE01 to your GitLab instance using ping or curl to rule out flaky network links.

• Misconfigured Runner Settings
  Double-check your config.toml for errors. For example, setting concurrent too high can overload the machine, causing random job failures. Ensure the shell executor uses a compatible shell (like bash) and that any custom environment variables aren’t causing conflicts.

• System Process Limits
  The runner might be hitting OS limits for open files or running processes. Check the gitlab-runner user’s ulimit settings with sudo -u gitlab-runner ulimit -a. If limits are low, adjust them in /etc/security/limits.conf and restart the runner service.

Start with upgrading the runner first—it’s the simplest fix and often resolves these vague system failure errors. If that doesn’t work, work through the other points one by one to narrow down the root cause.

内容的提问来源于stack exchange，提问作者Marcin K.