GitLab自动伸缩Docker Runner配置docker:dind遇连接问题及解决

初始环境与问题 #

我需要在GitLab CI中用TestContainers运行客户端库的测试套件，核心需求是从私有ECR镜像仓库拉取API实例镜像。按照TestContainers的推荐方案，要在CI任务中提前执行docker pull预拉取镜像。我们基于EC2搭建了自动伸缩的GitLab Docker+Machine Runner，日常运行正常，但拉取私有ECR镜像时遇到了两类问题：

Runner配置（config.toml） #

# 所有Runner可并发运行的任务上限
concurrent = 10
check_interval = 0

[[runners]]
  name = "Build-manager(Instance : i-REDACTED)"
  url = "https://our.gitlab.server/"
  token = "REDACTED"
  token_obtained_at = "REDACTED"
  token_expires_at = "REDACTED"
  executor = "docker+machine"
  # 该Runner可生成的最大机器数（运行中+闲置）
  limit = 5
  [runners.docker]
    image = "adoptopenjdk:11-jdk"
    privileged = true
    pull_policy = "if-not-present"
    tls_verify = false
    
    volumes = [
#              "/var/run/docker.sock:/var/run/docker.sock",
               "/cache",
               "/certs/client"
            ]
    
    # 禁用Docker执行器的内部缓存机制，改用分布式缓存
    disable_cache = true
    services_limit = -1
  [runners.cache]
    Type = "s3"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "s3.amazonaws.com"
      BucketName = "our-gitlab-cache-bucket"
      BucketLocation = "REDACTED"
  [runners.machine]
    IdleCount = 1
    IdleTime = 1800
    MaxBuilds = 100
    MachineDriver = "amazonec2"
    MachineName = "ci-build-runner-%s"
    MachineOptions = [
      "amazonec2-region=eu-west-1",
      "amazonec2-zone=a",
      "amazonec2-ami=REDACTED",
      "amazonec2-iam-instance-profile=Build-manager-runner-InstanceProfile",
      "amazonec2-vpc-id=REDACTED",
      "amazonec2-subnet-id=subnet-REDACTED",
      "amazonec2-private-address-only=true",
      "amazonec2-tags=CostId,Build-Runners,CostIdDetail,Build-Runners.Generic,InstanceType,Gitlab-runner,Project,Build-Runners,runner-manager-name,gitlab-aws-autoscaler",
      "amazonec2-instance-type=m5.xlarge",
      "amazonec2-security-group=Build-manager-runners_ACCESS",
      "amazonec2-request-spot-instance=true",
      "amazonec2-spot-price=0.1",
      "amazonec2-volume-type=gp3",
      "amazonec2-root-size=100",
      "amazonec2-userdata=/etc/gitlab-runner/runner-startup.sh",
      "amazonec2-volume-encrypted=true",
    ]
    [[runners.machine.autoscaling]]
      Periods = ["* * * * * mon-fri *"]
      IdleCount = 0
      IdleTime = 1800
      Timezone = "UTC"
    [[runners.machine.autoscaling]]
      Periods = ["* * 8-18 * * mon-fri *"]
      IdleCount = 0
      IdleTime = 3600
      Timezone = "UTC"
    [[runners.machine.autoscaling]]
      Periods = ["* * * * * sat,sun *"]
      IdleCount = 0
      IdleTime = 1800
      Timezone = "UTC"

.gitlab-ci.yml简化配置 #

default:
  image: adoptopenjdk:11-jdk
  services:
    - docker:dind

variables:
  # 指示TestContainers使用DinD的守护进程
  DOCKER_HOST: tcp://docker:2375
  # 使用overlayfs提升性能
  DOCKER_DRIVER: overlay2
  # 请保持该版本为最新
  API_VERSION: v2.1.1

test_only:
  stage: build
  script:
    # 安装docker等依赖
    - ./scripts/bash/install_os_dependencies.sh 
    # 安装aws-cli
    - ./scripts/bash/install_aws.sh 
#    - echo "PAUSING"
#    - sleep 600
    # 预拉取API镜像，供gradle构建中的TestContainers使用
    - ./scripts/bash/pre_fetch_docker_images.sh
    - ./scripts/bash/test.sh

预拉取镜像脚本（pre_fetch_docker_images.sh） #

#!/usr/bin/env bash
set -e

aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin $PRIVATE_ECR_REGISTRY_URL_BASE

# 若API_VERSION环境变量未定义则报错
: ${API_VERSION:?"Need to set API_VERSION"}

echo "**************************************************************************"
echo "* Using version $API_VERSION for tests"
echo "* Please make sure the environment variable API_VERSION is up to date!"
echo "**************************************************************************"
# 预拉取镜像，避免TestContainers自行拉取
docker pull $PRIVATE_ECR_REGISTRY_URL_BASE/api:$API_VERSION

具体故障表现 #

1. 不挂载docker.sock时：日志显示docker:dind服务已启动，但执行pre_fetch_docker_images.sh时，docker login成功，docker pull却提示无法连接到tcp://docker:2375的Docker守护进程。
2. 挂载docker.sock时：docker:dind服务启动失败，日志显示无法创建Unix Socket /var/run/docker.sock，提示"设备或资源忙"。此时任务执行时，docker login也失败，提示无法解析Docker主机。

解决过程与更新 #

更新1：添加DOCKER_TLS_CERTDIR配置 #

启动docker:dind容器前，日志提示容器缓存卷创建被禁用，无法创建/cache和/certs/client卷。尝试在.gitlab-ci.yml中添加DOCKER_TLS_CERTDIR: ""完全禁用TLS，配置如下：

default:
  image: adoptopenjdk:11-jdk
  services:
    - docker:dind

variables:
  # 指示TestContainers使用DinD的守护进程
  DOCKER_HOST: tcp://docker:2375
  # 禁用与docker守护进程的TLS通信
  DOCKER_TLS_CERTDIR: ""
  # 使用overlayfs提升性能
  DOCKER_DRIVER: overlay2

添加该配置后，所有功能恢复正常，但暂未明确具体原因。

更新2：明确无TLS DinD与Docker Socket的兼容性 #

在GitLab CI环境中，运行docker:dind与挂载主机docker.sock完全不兼容：前者会在容器内创建独立的Docker守护进程，后者则直接访问主机的Docker守护进程，二者只能二选一。感谢@pierrefaidherbe的提示，以及关于Docker 19.03版本强制DinD使用TLS的文档说明。

内容的提问来源于stack exchange，提问作者Luis Muñiz