GitLab流水线迁移至Kubernetes Executor替代dind+Compose方案咨询

我想把GitLab流水线从Docker executor切换为Kubernetes executor运行。目前我用Docker-in-Docker（dind）配合docker-compose部署Selenium网格及相关服务，有人建议用Kaniko替代，但我清楚Kaniko仅用于镜像构建，而我使用的是预构建镜像，且Kaniko不支持类似docker-compose的编排功能，因此不知道该如何适配Kubernetes executor，希望得到可行的解决方案。

以下是我的配置文件：

docker-compose.yml #

version: "3"
services:
  chrome:
    image: selenium/node-chrome:110.0
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443

  edge:
    image: selenium/node-edge:110.0
    shm_size: 2gb
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443

  firefox:
    image: selenium/node-firefox:110.0
    shm_size: 2gb
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443

  selenium-hub:
    image: selenium/hub:latest
    container_name: selenium-hub
    depends_on:
      - java-maven
    ports:
      - "4442:4442"
      - "4443:4443"
      - "4444:4444"

  java-maven:
    image: maven:3.8-openjdk-11

.gitlab-ci.yml #

variables:
  MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"

stages:
  - test
  
services:
  - docker:20.10.16-dind

test:
  stage: test
  image: docker:latest
  script:
    - docker-compose up -d
    - sleep 10
    - docker run --rm --network host -v "$(pwd)":/app -w /app maven:3.8-openjdk-11 mvn ${MAVEN_OPTS} test -Dcucumber.filter.tags=${FEATURE_TAG} 
  artifacts:
    when: always
    name: "report"
    paths:
      - test-output/*
    expire_in: 1 h

解决方案 #

1. 用GitLab CI的Kubernetes服务特性替代docker-compose #

Kubernetes executor允许在Job中定义多个sidecar服务，这些服务和主Job容器运行在同一个Pod内，可直接通过服务名或localhost访问，完全不需要dind和docker-compose。

修改后的.gitlab-ci.yml示例：

variables:
  MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"

stages:
  - test

test:
  stage: test
  image: maven:3.8-openjdk-11
  services:
    - name: selenium/hub:latest
      alias: selenium-hub
    - name: selenium/node-chrome:110.0
      alias: chrome
      variables:
        SE_EVENT_BUS_HOST: selenium-hub
        SE_EVENT_BUS_PUBLISH_PORT: 4442
        SE_EVENT_BUS_SUBSCRIBE_PORT: 4443
    - name: selenium/node-edge:110.0
      alias: edge
      variables:
        SE_EVENT_BUS_HOST: selenium-hub
        SE_EVENT_BUS_PUBLISH_PORT: 4442
        SE_EVENT_BUS_SUBSCRIBE_PORT: 4443
      shm_size: '2gb'
    - name: selenium/node-firefox:110.0
      alias: firefox
      variables:
        SE_EVENT_BUS_HOST: selenium-hub
        SE_EVENT_BUS_PUBLISH_PORT: 4442
        SE_EVENT_BUS_SUBSCRIBE_PORT: 4443
      shm_size: '2gb'
  script:
    # 用curl轮询替代sleep，确保Selenium网格就绪
    - until curl -s http://selenium-hub:4444/wd/hub/status | grep -q "ready"; do sleep 2; done
    - mvn ${MAVEN_OPTS} test -Dcucumber.filter.tags=${FEATURE_TAG} -Dselenium.grid.url=http://selenium-hub:4444/wd/hub
  artifacts:
    when: always
    name: "report"
    paths:
      - test-output/*
    expire_in: 1 h

关键说明：

• 主容器直接使用maven镜像，无需嵌套docker run操作，消除dind的权限和性能开销
• 每个Selenium组件作为sidecar服务定义，通过alias设置服务名，主容器可直接通过selenium-hub访问网格
• 需要在测试代码中指定Selenium网格地址为http://selenium-hub:4444/wd/hub

2. 将docker-compose转为Kubernetes Manifest，通过kubectl部署 #

如果需要更复杂的资源配置（如CPU/内存限制），可以把docker-compose转成K8s的Deployment/Service，再在CI中部署：

1. 用kompose工具转换docker-compose为K8s清单：

kompose convert -f docker-compose.yml

该命令会生成selenium-hub-deployment.yaml、chrome-deployment.yaml等文件。

2. 修改.gitlab-ci.yml完成部署、测试、清理流程：

variables:
  MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
  KUBE_NAMESPACE: "ci-test"

stages:
  - deploy
  - test
  - cleanup

deploy_selenium:
  stage: deploy
  image: bitnami/kubectl:latest
  script:
    - kubectl apply -f ./kubernetes-manifests/ -n $KUBE_NAMESPACE
    # 等待Selenium Hub部署就绪
    - kubectl wait --for=condition=available deployment/selenium-hub -n $KUBE_NAMESPACE --timeout=2m

test:
  stage: test
  image: maven:3.8-openjdk-11
  script:
    - mvn ${MAVEN_OPTS} test -Dcucumber.filter.tags=${FEATURE_TAG} -Dselenium.grid.url=http://selenium-hub.$KUBE_NAMESPACE.svc.cluster.local:4444/wd/hub
  artifacts:
    when: always
    name: "report"
    paths:
      - test-output/*
    expire_in: 1 h

cleanup_selenium:
  stage: cleanup
  image: bitnami/kubectl:latest
  when: always
  script:
    - kubectl delete -f ./kubernetes-manifests/ -n $KUBE_NAMESPACE

关键说明：

• 需要在GitLab中配置Kubernetes集群访问权限（通过GitLab集成K8s或设置KUBECONFIG变量）
• 部署到独立命名空间，测试完成后清理资源，避免集群资源浪费
• 测试代码通过K8s集群内部域名访问Selenium Hub

3. 使用Selenium官方Helm Chart部署（适合长期复用场景） #

如果集群支持Helm，可以直接用Selenium官方Helm Chart快速部署标准化的网格：

variables:
  MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
  KUBE_NAMESPACE: "ci-test"

stages:
  - deploy
  - test
  - cleanup

deploy_selenium:
  stage: deploy
  image: alpine/helm:latest
  script:
    - helm repo add selenium https://seleniumhq.github.io/docker-selenium
    - helm install selenium-grid selenium/selenium-grid -n $KUBE_NAMESPACE --set chromeNode.replicas=1,firefoxNode.replicas=1,edgeNode.replicas=1
    - kubectl wait --for=condition=available deployment/selenium-grid-hub -n $KUBE_NAMESPACE --timeout=2m

test:
  stage: test
  image: maven:3.8-openjdk-11
  script:
    - mvn ${MAVEN_OPTS} test -Dcucumber.filter.tags=${FEATURE_TAG} -Dselenium.grid.url=http://selenium-grid-hub.$KUBE_NAMESPACE.svc.cluster.local:4444/wd/hub
  artifacts:
    when: always
    name: "report"
    paths:
      - test-output/*
    expire_in: 1 h

cleanup_selenium:
  stage: cleanup
  image: alpine/helm:latest
  when: always
  script:
    - helm uninstall selenium-grid -n $KUBE_NAMESPACE

关键说明：

• 通过Helm可以快速配置节点数量、资源限制等参数
• 官方Chart维护更规范，适合需要频繁复用Selenium网格的场景

内容的提问来源于stack exchange，提问作者SRM21