GitLab Webhook测试返回200但服务未触发脚本的排查

问题描述 #

GitLab配置Webhook后，测试触发返回状态码200，但目标机器上的systemd监听服务未执行构建脚本（脚本直接运行正常），需排查并修复Webhook服务接收GitLab通知的配置。

现有配置信息 #

1. Ansible配置的systemd服务 #

- name: Create the webhooks service
  copy:
    dest: /etc/systemd/system/webhook.service
    content: |
      [Unit]
      Description=Webhooks
      [Service]
      ExecStart=/usr/bin/webhook -secure \
      -cert /etc/letsencrypt/live/my-project.mywebsite.com/fullchain.pem \
      -key /etc/letsencrypt/live/my-project.mywebsite.com/privkey.pem \
      -hooks /home/myuser/webhooks/hooks.json \
      -hotreload \
      -verbose
      WorkingDirectory=/home/myuser/webhooks
      Restart=always
      RestartSec=5
      [Install]
      WantedBy=multi-user.target

- name: Enable the webhooks service
  command: systemctl enable webhook.service

- name: (Re-)start the webhooks service
  command: systemctl restart webhook.service

2. hooks.json配置 #

[{
  "id": "my-repository",
  "execute-command": "/home/myuser/webhooks/my-repository/deploy.sh",
  "command-working-directory": "/home/myuser/my-repository/",
  "response-message": "Executing deploy script...",
  "trigger-rule": {
    "match": {
      "type": "payload-hmac-sha1",
      "secret":"<WEBHOOK_SECRET_LIKE_IN_GITLAB>",
      "parameter": {
        "source": "header",
        "name": "X-Hub-Signature"
      }
    }
  }
}]

3. 端口监听确认 #

myuser@mymachine:~$ sudo lsof -i -P -n | grep 9000
webhook   609262            root    6u  IPv6 1589700      0t0  TCP *:9000 (LISTEN)

4. systemd服务状态 #

myuser@mymachine:~$ sudo systemctl status webhook.service
● webhook.service - Webhooks
     Loaded: loaded (/etc/systemd/system/webhook.service; enabled; preset: enabled)
     Active: active (running) since Tue 2024-10-08 18:42:34 UTC; 23s ago
   Main PID: 637049 (webhook)
      Tasks: 9 (limit: 18692)
     Memory: 2.1M (peak: 2.4M)
        CPU: 8ms
     CGroup: /system.slice/webhook.service
             └─637049 /usr/bin/webhook -secure -cert /etc/letsencrypt/live/my-project.mywebsite.com/fullchain.pem -key /etc/letsencrypt/live/my-project.mywebsite.com/privkey.pem -hooks >

Oct 08 18:42:34 my-project.mywebsite.com systemd[1]: Started webhook.service - Webhooks.
Oct 08 18:42:34 my-project.mywebsite.com webhook[637049]: [webhook] 2024/10/08 18:42:34 version 2.8.0 starting
Oct 08 18:42:34 my-project.mywebsite.com webhook[637049]: [webhook] 2024/10/08 18:42:34 setting up os signal watcher
Oct 08 18:42:34 my-project.mywebsite.com webhook[637049]: [webhook] 2024/10/08 18:42:34 attempting to load hooks from /home/myuser/webhooks/hooks.json
Oct 08 18:42:34 my-project.mywebsite.com webhook[637049]: [webhook] 2024/10/08 18:42:34 found 1 hook(s) in file
Oct 08 18:42:34 my-project.mywebsite.com webhook[637049]: [webhook] 2024/10/08 18:42:34         loaded: my-repository
Oct 08 18:42:34 my-project.mywebsite.com webhook[637049]: [webhook] 2024/10/08 18:42:34 os signal watcher ready
Oct 08 18:42:34 my-project.mywebsite.com webhook[637049]: [webhook] 2024/10/08 18:42:34 setting up file watcher for /home/myuser/webhooks/hooks.json
Oct 08 18:42:34 my-project.mywebsite.com webhook[637049]: [webhook] 2024/10/08 18:42:34 serving hooks on https://0.0.0.0:9000/hooks/{id}

网络环境 #

GitLab服务器与目标机器位于同一Hetzner私有网络，GitLab地址为10.0.2.1，目标机器地址为10.0.3.1。

排查与解决步骤 #

1. 修复HMAC签名匹配规则 #

GitLab发送的X-Hub-Signature格式为sha1=xxxxxx，webhook工具默认不会自动识别前缀，需在hooks.json的触发规则中添加prefix字段：

"trigger-rule": {
  "match": {
    "type": "payload-hmac-sha1",
    "secret":"<WEBHOOK_SECRET_LIKE_IN_GITLAB>",
    "parameter": {
      "source": "header",
      "name": "X-Hub-Signature"
    },
    "prefix": "sha1="
  }
}

修改后重启服务：sudo systemctl restart webhook.service

2. 确认权限配置 #

当前webhook以root用户运行，需确保root能访问并执行脚本：

• 检查脚本执行权限：chmod +x /home/myuser/webhooks/my-repository/deploy.sh
• 检查工作目录权限：确保root对/home/myuser/my-repository/有读写权限

3. 查看详细日志定位问题 #

利用-verbose参数查看实时日志：

journalctl -u webhook.service -f

触发GitLab测试Webhook，观察日志中是否有签名验证失败、脚本执行报错等信息，精准定位问题。

4. 模拟请求测试 #

在目标机器上用curl模拟GitLab请求，验证签名和触发逻辑：

# 生成测试payload的签名
echo -n "test payload" | openssl dgst -sha1 -hmac "<WEBHOOK_SECRET_LIKE_IN_GITLAB>" | sed 's/^.* //'
# 发送请求
curl -X POST https://10.0.3.1:9000/hooks/my-repository \
  -H "X-Hub-Signature: sha1=生成的签名值" \
  -H "Content-Type: application/json" \
  -d '{"test": "payload"}'

观察是否触发脚本执行，同时核对日志输出。

5. 排查HTTPS证书问题（私有网络场景） #

若私有网络内无需HTTPS，可暂时禁用HTTPS测试：
修改systemd服务的ExecStart：

ExecStart=/usr/bin/webhook -hooks /home/myuser/webhooks/hooks.json -hotreload -verbose

重启服务后，将GitLab的Webhook地址改为http://10.0.3.1:9000/hooks/my-repository测试。若正常执行，再排查GitLab是否不信任目标机器的Let's Encrypt证书。

内容的提问来源于stack exchange，提问作者C4X