GitLab-workhorse缺失socket致服务宕机，请求排查解决方法

问题现象 #

• 配置GitLab+Nginx反向代理后，页面加载返回502错误
• 执行gitlab-ctl status查看服务状态，结果显示gitlab-workhorse异常：

  down: gitlab-workhorse: 0s, normally up, want up; run: log: (pid 3756258) 12450s
  

• 查看/var/opt/gitlab/gitlab-workhorse目录（执行ls -ap /var/opt/gitlab/gitlab-workhorse/），仅包含：

  ./  ../  config.toml  VERSION
  

  目录下无socket文件，且缺失配置中指定的sockets子目录

配置信息 #

GitLab主配置文件（/etc/gitlab/gitlab.rb）相关内容：

nginx['enable'] = false
web_server['external_users'] = ['www-data']
gitlab_rails['trusted_proxies'] = ['127.0.0.1', <external-server-ip>]
gitlab_workhorse['listen_network'] = "unix"
gitlab_workhorse['listen_addr'] = "/var/opt/gitlab/gitlab-workhorse/sockets/socket"

日志信息 #

1. Nginx错误日志：

   connect() to unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket failed (13: Permission denied) while connecting to upstream
   

2. GitLab Workhorse日志（/var/log/gitlab/gitlab-workhorse/current）：

   {"build_time":"20221024.191252","level":"info","msg":"Starting","time":"2022-10-30T20:05:21+03:00","version":"v15.5.1"}
   {"address":"localhost:9229","level":"info","msg":"Running metrics server","network":"tcp","time":"2022-10-30T20:05:21+03:00"}
   {"level":"info","msg":"keywatcher: starting process loop","time":"2022-10-30T20:05:21+03:00"}
   {"address":"/var/opt/gitlab/redis/redis.socket","level":"info","msg":"redis: dialing","network":"unix","time":"2022-10-30T20:05:21+03:00"}
   {"address":"/var/opt/gitlab/gitlab-workhorse/sockets/socket","level":"info","msg":"Running upstream server","network":"unix","time":"2022-10-30T20:05:21+03:00"}
   {"error":"listen unix /var/opt/gitlab/gitlab-workhorse/sockets/socket: bind: no such file or directory","level":"fatal","msg":"shutting down","time":"2022-10-30T20:05:21+03:00"}
   

核心原因与解决步骤 #

问题根源 #

从Workhorse的致命错误日志可直接定位：配置中指定的socket文件目录/var/opt/gitlab/gitlab-workhorse/sockets/不存在，导致Workhorse无法创建socket文件并启动，最终引发Nginx的502错误。

解决步骤 #

1. 创建缺失的sockets目录
   手动创建目录并设置正确的权限（归属git用户组）：

   sudo mkdir -p /var/opt/gitlab/gitlab-workhorse/sockets
   sudo chown git:git /var/opt/gitlab/gitlab-workhorse/sockets
   sudo chmod 755 /var/opt/gitlab/gitlab-workhorse/sockets
   

2. 重新配置并重启GitLab服务
   应用配置变更并重启Workhorse：

   sudo gitlab-ctl reconfigure
   sudo gitlab-ctl restart gitlab-workhorse
   

3. 验证服务状态
   再次执行gitlab-ctl status确认gitlab-workhorse状态为up；同时检查sockets目录是否生成了socket文件：

   ls -l /var/opt/gitlab/gitlab-workhorse/sockets/
   

4. 修复Nginx权限问题（可选）
   如果后续仍出现Nginx的权限拒绝错误，需确保www-data用户对socket文件有访问权限：

   sudo chmod 770 /var/opt/gitlab/gitlab-workhorse/sockets/socket
   sudo usermod -aG git www-data
   

   执行后重启Nginx：

   sudo systemctl restart nginx
   

内容的提问来源于stack exchange，提问作者Skomich