GitLab Pipeline执行时导致Kubernetes集群Pod崩溃的原因排查

我的Kubernetes集群中运行着GitLab-ee 15.8.0和GitLab Runner（15.8.2），Runner配置为kubernetes executor，并且在ConfigMap中挂载了宿主机的/var/run/docker.sock。当执行包含启动docker-compose-test.yml的Pipeline时，集群中所有Pod开始崩溃重启，之后Pipeline一直处于Running状态但无后续处理，最后执行的命令是docker-compose -f docker-compose-test.yml up -d。

预期Pipeline仅启动Docker容器，通过数据库和应用容器运行Laravel测试，但实际破坏了Nginx-Ingress资源。

环境版本 #

• GitLab-ee: 15.8.0
• GitLab Runner: 15.8.2

相关配置文件 #

gitlab-ci.yml #

image: docker:20.10.16

services:
  - docker:20.10.16-dind

variables:
  DOCKER_COMPOSE_CMD: "docker-compose -f docker-compose-test.yml"
  

stages:
  - test
  - build
  

test:
  stage: test
  script:
    - docker-compose --version
    - $DOCKER_COMPOSE_CMD down --volumes --remove-orphans
    - $DOCKER_COMPOSE_CMD up -d
    - $DOCKER_COMPOSE_CMD exec -T -e APP_ENV=testing laravel-api-test ./scripts/wait-for.sh database-test:54321 -t 60 -- echo "Database connection established"
    - $DOCKER_COMPOSE_CMD exec -T -e APP_ENV=testing laravel-api-test php artisan passport:keys
    - $DOCKER_COMPOSE_CMD exec -T -e APP_ENV=testing laravel-api-test php artisan migrate
    - $DOCKER_COMPOSE_CMD exec -T -e APP_ENV=testing laravel-api-test sh -c "vendor/bin/phpunit ./tests $PARAMETERS --coverage-text --colors=never --stderr"
    - $DOCKER_COMPOSE_CMD down --volumes --remove-orphans
  # only:
  #   - tags

build:
  stage: build
  script:
    - export IMAGE_TAG=$(echo "$CI_COMMIT_TAG" | awk -F '/' '{print $NF}')
    - docker build -t laravel-api:"$IMAGE_TAG" .
    - docker login -u "$CONTAINER_REGISTRY_USERNAME" -p "$CONTAINER_REGISTRY_PASSWORD" "$CONTAINER_REGISTRY_URL"
    - docker push laravel-api:"$IMAGE_TAG"
  only:
    - tags

docker-compose-test.yml #

version: "3.7"
services:
  laravel-api-test:
    build:
      args:
        user: laravel
        uid: 1000
      context: .
      dockerfile: docker/development/Dockerfile
    working_dir: /var/www/
    volumes:
    - ./:/var/www
    ports:
    - ${APP_PORT}:9000
    networks:
    - application

  database-test:
    image: postgres:15.1-alpine
    ports:
    - 54321:5432
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
    networks:
    - application

networks:
  application:
    driver: bridge

GitLab Runner ConfigMap #

apiVersion: v1
kind: ConfigMap
metadata:
  name: gitlab-runner-config
  namespace: gitlab-runner
data:
  config.toml: |-
    concurrent = 4
    [[runners]]
      name = "Runner_1"
      url = "https://gitlab.project.com/ci"
      token = "my-token"
      executor = "kubernetes"
      [runners.kubernetes]
        namespace = "gitlab-runner"
        privileged = true
        poll_timeout = 600
        cpu_request = "1"
        service_cpu_request = "200m"
        [[runners.kubernetes.volumes.host_path]]
            name = "docker"
            mount_path = "/var/run/docker.sock"
            host_path = "/var/run/docker.sock"

Pipeline崩溃后输出 #

Running with gitlab-runner 15.8.2 (4d1ca121)

on Runner_1 eNNz4y9k, system ID: r_y3jEhmF8fN58

Preparing the "kubernetes" executor

00:00

Using Kubernetes namespace: gitlab-runner

Using Kubernetes executor with image docker:20.10.16 ...

Using attach strategy to execute scripts...

Preparing environment

00:04

Waiting for pod gitlab-runner/runner-ennz4y9k-project-117-concurrent-0f24cx to be running, status is Pending

Running on runner-ennz4y9k-project-117-concurrent-0f24cx via gitlab-runner-56cd6f4bb5-zrbd9...

Getting source from Git repository

00:01

Fetching changes with git depth set to 20...

Initialized empty Git repository in /builds/Clients/opus-volvere/laravel-api/.git/

Created fresh repository.

Checking out 3890412c as main...

Skipping Git submodules setup

Executing "step_script" stage of the job script

$ docker-compose --version

Docker Compose version v2.6.0

$ $DOCKER_COMPOSE_CMD down --volumes --remove-orphans

Container laravel-api-database-test-1 Stopping

Container laravel-api-laravel-api-test-1 Stopping

Container laravel-api-database-test-1 Stopping

Container laravel-api-laravel-api-test-1 Stopping

Container laravel-api-database-test-1 Stopped

Container laravel-api-database-test-1 Removing

Container laravel-api-laravel-api-test-1 Stopped

Container laravel-api-laravel-api-test-1 Removing

Container laravel-api-laravel-api-test-1 Removed

Container laravel-api-database-test-1 Removed

Network laravel-api_application Removing

Network laravel-api_application Removed

$ $DOCKER_COMPOSE_CMD up -d

#1 [internal] load build definition from Dockerfile

#1 transferring dockerfile: 827B done

#1 DONE 0.1s

#2 [internal] load .dockerignore

#2 transferring context: 88B done

#2 DONE 0.1s

不确定从何处排查，希望得到调试和解决方法的帮助。异常仅在启动docker-compose时发生，之前构建镜像步骤正常，猜测可能与Runner的Pod权限有关。

核心问题出在GitLab Runner挂载了宿主机的/var/run/docker.sock：

1. gitlab-ci.yml中配置了docker:20.10.16-dind服务，预期让Docker命令操作DinD（Docker-in-Docker）容器内的Docker daemon，但同时Runner挂载了宿主机的docker.sock，导致所有docker-compose命令直接操作宿主机的Docker daemon，而非DinD容器内的。
2. 执行docker-compose up -d时，容器直接创建在Kubernetes节点的宿主机上，而非隔离的DinD环境中：
   • 如果Kubernetes集群使用Docker作为容器运行时，宿主机的Docker daemon同时管理Kubernetes的Pod容器和测试容器，docker-compose的操作会干扰Kubernetes组件。
   • docker-compose创建的网络、端口映射可能与Nginx-Ingress等集群组件的资源冲突；加上Runner Pod是privileged模式，权限过高，甚至可能误删Kubernetes系统容器/卷，直接导致集群Pod崩溃。
   • docker-compose down --volumes --remove-orphans操作宿主机Docker daemon时，可能误删除Kubernetes相关的匿名卷或容器，破坏集群稳定性。

1. 移除宿主机docker.sock的挂载 #

修改GitLab Runner的ConfigMap，删除挂载宿主机docker.sock的配置，让Runner使用DinD服务的Docker daemon：

apiVersion: v1
kind: ConfigMap
metadata:
  name: gitlab-runner-config
  namespace: gitlab-runner
data:
  config.toml: |-
    concurrent = 4
    [[runners]]
      name = "Runner_1"
      url = "https://gitlab.project.com/ci"
      token = "my-token"
      executor = "kubernetes"
      [runners.kubernetes]
        namespace = "gitlab-runner"
        privileged = true
        poll_timeout = 600
        cpu_request = "1"
        service_cpu_request = "200m"
    # 移除宿主机docker.sock挂载配置
    # [[runners.kubernetes.volumes.host_path]]
    #     name = "docker"
    #     mount_path = "/var/run/docker.sock"
    #     host_path = "/var/run/docker.sock"

2. 配置环境变量指向DinD服务 #

在gitlab-ci.yml中添加环境变量，确保docker和docker-compose命令使用DinD容器的Docker daemon：

image: docker:20.10.16

services:
  - docker:20.10.16-dind

variables:
  DOCKER_COMPOSE_CMD: "docker-compose -f docker-compose-test.yml"
  # 添加以下环境变量
  DOCKER_HOST: tcp://docker:2375
  DOCKER_TLS_CERTDIR: ""

3. 调整docker-compose配置（可选） #

由于使用DinD环境，测试容器仅需内部访问，可移除宿主机端口映射，避免不必要的资源冲突：

version: "3.7"
services:
  laravel-api-test:
    build:
      args:
        user: laravel
        uid: 1000
      context: .
      dockerfile: docker/development/Dockerfile
    working_dir: /var/www/
    volumes:
    - ./:/var/www
    # 移除宿主机端口映射
    # ports:
    # - ${APP_PORT}:9000
    networks:
    - application

  database-test:
    image: postgres:15.1-alpine
    # 移除宿主机端口映射
    # ports:
    # - 54321:5432
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
    networks:
    - application

networks:
  application:
    driver: bridge

若问题仍存在，可通过以下步骤排查：

• 查看GitLab Runner Pod日志：kubectl logs -n gitlab-runner <gitlab-runner-pod-name>，检查Docker连接或权限错误。
• 查看Pipeline Pod日志：在GitLab Pipeline详情中找到对应Pod名称，执行kubectl logs -n gitlab-runner <pipeline-pod-name>，查看docker-compose命令的详细错误输出。
• 查看Kubernetes节点系统日志：journalctl -u docker.service（使用Docker运行时），检查宿主机Docker daemon的异常操作或错误。

内容的提问来源于stack exchange，提问作者A. Vreeswijk