GitLab CE与GitLab Kubernetes Agent Server连接失败求助

在DigitalOcean Kubernetes集群中配置GitLab Agent，尝试连接NGINX反向代理后的GitLab CE 15.4 Omnibus实例时失败。

安装命令 #

使用Helm安装GitLab Agent的命令：

helm upgrade --install my-cluster gitlab/gitlab-agent --namespace gitlab-agent --create-namespace --set image.tag=v15.4.0 --set config.token=<token> --set config.kasAddress=wss://<my-instance>/-/kubernetes-agent/

错误日志 #

Agent日志出现以下错误：

{"level":"error","time":"2022-11-06T18:23:37.960Z","msg":"Error handling a connection","mod_name":"reverse_tunnel","error":"Connect(): rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: expected handshake response status code 101 but got 302\""}

当前NGINX配置 #

upstream gitlab-workhorse {
  # On GitLab versions before 13.5, the location is
  # `/var/opt/gitlab/gitlab-workhorse/socket`. Change the following line
  # accordingly.
  server unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket fail_timeout=0;
}

server {
  listen 0.0.0.0:80;
  listen [::]:80 ipv6only=on;
  server_name  <my-domain>;
  server_tokens off;

  return 301 https://$http_host$request_uri;   
  
# individual nginx logs for this gitlab vhost
  access_log  /var/log/nginx/gitlab_access.log;
  error_log   /var/log/nginx/gitlab_error.log;
}

## HTTPS host
server {
  listen 0.0.0.0:443 ssl;
  listen [::]:443 ipv6only=on ssl;
  server_name <my-domain>;
  server_tokens off; ## Don't show the nginx version number, a security best practice
  root /opt/gitlab/embedded/service/gitlab-rails/public;

  ## Strong SSL Security
  ## https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html & https://cipherli.st/
    ssl_certificate /etc/letsencrypt/live/<my-domain>/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/<my-domain>/privkey.pem; # managed by Certbot
  include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot


  # GitLab needs backwards compatible ciphers to retain compatibility with Java IDEs
  #ssl_ciphers "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
  #ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  #ssl_prefer_server_ciphers on;
  #ssl_session_cache shared:SSL:10m;
  #ssl_session_timeout 5m;

  ## See app/controllers/application_controller.rb for headers set

  ## [Optional] Enable HTTP Strict Transport Security
  ## HSTS is a feature improving protection against MITM attacks
  ## For more information see: https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/
  # add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";

  ## [Optional] If your certficate has OCSP, enable OCSP stapling to reduce the overhead and latency of running SSL.
  ## Replace with your ssl_trusted_certificate. For more info see:
  ## - https://medium.com/devops-programming/4445f4862461
  ## - https://www.ruby-forum.com/topic/4419319
  ## - https://www.digitalocean.com/community/tutorials/how-to-configure-ocsp-stapling-on-apache-and-nginx
  # ssl_stapling on;
  # ssl_stapling_verify on;
  # ssl_trusted_certificate /etc/nginx/ssl/stapling.trusted.crt;
  # resolver 208.67.222.222 208.67.222.220 valid=300s; # Can change to your DNS resolver if desired
  # resolver_timeout 5s;

  ## [Optional] Generate a stronger DHE parameter:
  ##   sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 4096
  ##
  # ssl_dhparam /etc/ssl/certs/dhparam.pem;

  ## Individual nginx logs for this GitLab vhost
  access_log  /var/log/nginx/gitlab_access.log;
  error_log   /var/log/nginx/gitlab_error.log;

  location / {
    client_max_body_size 0;
    gzip off;

    ## https://github.com/gitlabhq/gitlabhq/issues/694
    ## Some requests take more than 30 seconds.
    proxy_read_timeout      300;
    proxy_connect_timeout   300;
    proxy_redirect          off;

    proxy_http_version 1.1;

    proxy_set_header    Host                $http_host;
    proxy_set_header    X-Real-IP           $remote_addr;
    proxy_set_header    X-Forwarded-Ssl     on;
    proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
    proxy_set_header    X-Forwarded-Proto   $scheme;

    proxy_pass http://gitlab-workhorse;
  }
}

已尝试的解决方法 #

• 启用GitLab KAS：

  gitlab_kas['enable'] = true
  

• 在NGINX配置中添加专属location块：

  location /-/kubernetes-agent/ {
    proxy_pass http://gitlab-workhorse;
    proxy_http_version 1.1;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-NginX-Proxy true;
    proxy_set_header Host $host;
    proxy_set_header Sec-WebSocket-Protocol $http_sec_websocket_protocol;
    proxy_set_header Sec-WebSocket-Extensions $http_sec_websocket_extensions;
    proxy_set_header Sec-WebSocket-Key $http_sec_websocket_key;
    proxy_set_header Sec-WebSocket-Version $http_sec_websocket_version;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_cache_bypass $http_upgrade;
  }
  

• 使用ws://替代wss://作为KAS地址

解决方案 #

1. 修正NGINX的WebSocket代理配置 #

问题出在WebSocket握手时返回302，说明NGINX未正确处理代理转发，或者GitLab KAS路由配置未生效：

• 在NGINX配置的server块外部添加map指令处理WebSocket的Connection头：

  map $http_upgrade $connection_upgrade {
      default upgrade;
      '' close;
  }
  

• 调整/-/kubernetes-agent/的location块，确保proxy_pass末尾不带斜杠（避免路径重写问题），并补充超时设置：

  location /-/kubernetes-agent/ {
      proxy_pass http://gitlab-workhorse;
      proxy_http_version 1.1;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;
      proxy_read_timeout 86400;
      proxy_cache_bypass $http_upgrade;
  }
  

修改后重启NGINX服务。

2. 完善GitLab KAS的Omnibus配置 #

仅启用KAS不够，需在/etc/gitlab/gitlab.rb中补充完整配置：

gitlab_kas['enable'] = true
gitlab_kas['external_url'] = 'wss://<my-domain>/-/kubernetes-agent/'
gitlab_kas['listen_address'] = '0.0.0.0:8150'
gitlab_workhorse['gitlab_kas_url'] = 'http://localhost:8150'

执行gitlab-ctl reconfigure和gitlab-ctl restart使配置生效。

3. 修正Helm安装参数 #

确保kasAddress与GitLab配置的外部URL完全一致，末尾不要多余斜杠：

helm upgrade --install my-cluster gitlab/gitlab-agent --namespace gitlab-agent --create-namespace --set image.tag=v15.4.0 --set config.token=<token> --set config.kasAddress=wss://<my-domain>/-/kubernetes-agent

4. 验证连接与路由 #

• 查看NGINX访问日志，确认/-/kubernetes-agent/的请求未返回302
• 在GitLab服务器上测试WebSocket握手：

  curl -i -N -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Sec-WebSocket-Key: test" -H "Sec-WebSocket-Version: 13" https://<my-domain>/-/kubernetes-agent/
  

预期返回101状态码，说明握手成功。

内容的提问来源于stack exchange，提问作者Bram Hurkmans