GitLab CI中使用docker:dind无法连接容器内localhost的问题

问题 #

本地执行以下requests调用可正常工作：

res = requests.post(f"{API_BASE_URL}/dags/sql_data_transfer/dagRuns", headers={
    'Content-Type': 'application/json',
    'Authorization': f'Bearer {TEST_TOKEN}'},

                    json={
                        "dag_run_id": dag_run_id,
                        'conf': conf
                    }
                    )

但在GitLab CI环境使用docker:dind时，无论用localhost、127.0.0.1还是docker作为主机，均无法连接，报错如下：

if isinstance(e.reason, _ProxyError):
                raise ProxyError(e, request=request)
    
            if isinstance(e.reason, _SSLError):
                # This branch is for urllib3 v1.22 and later.
                raise SSLError(e, request=request)
    
>           raise ConnectionError(e, request=request)
E           requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8080): Max retries exceeded with url: /api/v1/dags/sql_data_transfer/dagRuns (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8e98491bd0>: Failed to establish a new connection: [Errno 111] Connection refused'))
/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py:519: ConnectionError

当前docker-compose.yml配置：

version: '2.2'
x-airflow-common:
  &airflow-common
  image: airflow
  build: airflow
  environment:
    - AIRFLOW__CORE__EXECUTOR=LocalExecutor #
    - AIRFLOW__CORE__LOAD_EXAMPLES=False
    - AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION=True
    - AIRFLOW__LOGGING__LOGGING_LEVEL=INFO


  volumes:
    - ./airflow/dags:/opt/airflow/dags
    - ./airflow/airflow-data/logs:/opt/airflow/logs
    - ./airflow/airflow-data/plugins:/opt/airflow/plugins
    - ./airflow/config/airflow.cfg:/opt/airflow/airflow.cfg
  user: "${AIRFLOW_UID:-50000}:0"
  depends_on:
    - postgres

services:
  airflow-observer:
    << : *airflow-common
    container_name: airflow_observer
    command: airflow webserver
    ports:
      - 8082:8080

解决方案 #

1. 修正GitLab CI中的服务访问主机名 #

在docker:dind环境中，CI任务容器和Airflow容器属于同一Docker网络，localhost指向CI任务自身容器，无法访问Airflow服务。需使用服务名称作为主机：

• 将API_BASE_URL设置为http://airflow-observer:8080
• 端口用容器内部的8080，而非宿主机映射的8082，容器间通信直接使用内部端口

2. 规范GitLab CI的Docker环境配置 #

确保.gitlab-ci.yml正确配置docker:dind服务，示例片段：

variables:
  DOCKER_HOST: tcp://docker:2375/
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ""

services:
  - docker:dind

test_airflow_api:
  stage: test
  image: docker:latest
  before_script:
    - apk add --no-cache docker-compose python3 py3-pip
    - pip install requests
  script:
    - docker-compose up -d
    - sleep 60  # 或用健康检查等待服务就绪
    - python your_request_script.py

3. 给Airflow服务添加健康检查 #

避免在服务未完全启动时发起请求，修改docker-compose.yml的airflow-observer服务：

services:
  airflow-observer:
    << : *airflow-common
    container_name: airflow_observer
    command: airflow webserver
    ports:
      - 8082:8080
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 5

然后在CI脚本中用docker-compose wait airflow-observer等待服务健康后再执行请求。

4. 确认Airflow API的启用状态 #

检查airflow.cfg配置：

• 确保api_enabled = True
• 确认API端口为8080（默认值）

内容的提问来源于stack exchange，提问作者Assem