Docker部署Airflow：自动启动服务并解决初始化依赖问题

我尝试用Docker部署Airflow，单容器部署时每次启动都得手动进入容器执行airflow scheduler，想实现自动化启动。改用初始化、webserver、scheduler三容器方案后失败，原因是docker-compose的depends_on只保证容器启动顺序，不会等待初始化容器的airflow db init执行完成，导致webserver和scheduler启动时报错要求执行airflow db init。

三容器部署的docker-compose.yml #

version: '3.8'
services:
  postgres:
    image: postgres:15.3
    environment:
      POSTGRES_USER: admin
      POSTGRES_PASSWORD: admin
      POSTGRES_DB: postgres_db
    volumes:
      - ./postgres_data:/var/lib/postgresql/data
    ports:
      - "5432:5432"
    networks:
      - mynetwork
      
  airflow-init:
    build: ./airflow
    command: >
      bash -c "
        airflow db init && 
        airflow users create --username admin --password admin --firstname Admin --lastname User --role Admin --email admin@example.com
      "
    volumes:
      - ./airflow/dags:/opt/airflow/dags
      - ./csv data:/opt/csv_data
    depends_on:
      - postgres
    networks:
      - mynetwork
      
  airflow-webserver:
    build: ./airflow
    command: airflow webserver
    volumes:
      - ./airflow/dags:/opt/airflow/dags
      - ./csv data:/opt/csv_data
    ports:
      - 8080:8080
    depends_on:
      - airflow-init
      - postgres
    networks:
      - mynetwork

  airflow-scheduler:
    build: ./airflow
    command: airflow scheduler
    volumes:
      - ./airflow/dags:/opt/airflow/dags
      - ./csv data:/opt/csv_data
    depends_on:
      - airflow-init
      - postgres
    networks:
      - mynetwork

networks:
  mynetwork:
    driver: bridge

可运行但需手动启动scheduler的单容器docker-compose.yml #

version: '3.8'
services:
  postgres:
    image: postgres:15.3
    environment:
      POSTGRES_USER: admin
      POSTGRES_PASSWORD: admin
      POSTGRES_DB: postgres_db
    volumes:
      - ./postgres_data:/var/lib/postgresql/data
    ports:
      - "5432:5432"
    networks:
      - mynetwork
         
  airflow:
    build: ./airflow
    command: >
      bash -c "
        airflow db init && 
        airflow users create --username admin --password admin --firstname Admin --lastname User --role Admin --email admin@example.com &&
        airflow webserver &&
        airflow scheduler
      "
    volumes:
      - ./airflow/dags:/opt/airflow/dags
      - ./csv data:/opt/csv_data
    ports:
      - 8080:8080
    depends_on:
      - postgres
    networks:
      - mynetwork

networks:
  mynetwork:
    driver: bridge

错误日志 #

scheduler容器日志 #

2023-07-02 19:34:42 ERROR: You need to initialize the database. Please run `airflow db init`. Make sure the command is run using Airflow version 2.6.2.

webserver容器日志 #

2023-07-02 19:34:40 ERROR: You need to initialize the database. Please run `airflow db init`. Make sure the command is run using Airflow version 2.6.2.

补充说明：默认airflow:airflow用户无法使用，已创建Admin用户；尝试过官方CeleryExecutor配置脚本但未成功。

解决方案 #

方案1：给webserver和scheduler添加启动等待脚本 #

由于depends_on不等待初始化任务完成，我们可以在webserver和scheduler的启动命令前加入等待逻辑，检查数据库是否已初始化完成。

修改airflow-webserver和airflow-scheduler的command：

airflow-webserver:
  build: ./airflow
  command: >
    bash -c "
      while ! airflow db check; do
        echo '等待数据库初始化完成...'
        sleep 5
      done
      airflow webserver
    "
  # 其余配置保持不变

airflow-scheduler:
  build: ./airflow
  command: >
    bash -c "
      while ! airflow db check; do
        echo '等待数据库初始化完成...'
        sleep 5
      done
      airflow scheduler
    "
  # 其余配置保持不变

airflow db check会验证数据库连接和初始化状态，循环执行直到成功，再启动对应服务。

方案2：使用初始化容器的健康状态判断 #

给airflow-init容器添加healthcheck，让Docker判断初始化任务是否完成，然后让webserver和scheduler依赖于初始化容器的健康状态。

首先修改airflow-init配置：

airflow-init:
  build: ./airflow
  command: >
    bash -c "
      airflow db init && 
      airflow users create --username admin --password admin --firstname Admin --lastname User --role Admin --email admin@example.com
    "
  volumes:
    - ./airflow/dags:/opt/airflow/dags
    - ./csv data:/opt/csv_data
  depends_on:
    - postgres
  networks:
    - mynetwork
  healthcheck:
    test: ["CMD-SHELL", "airflow db check"]
    interval: 5s
    timeout: 10s
    retries: 5

然后修改webserver和scheduler的depends_on规则：

airflow-webserver:
  # 其余配置保持不变
  depends_on:
    postgres:
      condition: service_started
    airflow-init:
      condition: service_healthy

airflow-scheduler:
  # 其余配置保持不变
  depends_on:
    postgres:
      condition: service_started
    airflow-init:
      condition: service_healthy

这样Docker会等待airflow-init容器进入健康状态（即数据库初始化完成），才启动webserver和scheduler。

方案3：优化单容器部署（无需手动启动scheduler） #

如果想继续使用单容器，原配置中airflow webserver是前台运行的，会阻塞后续的airflow scheduler执行。可以把scheduler放到后台运行：

修改单容器的command：

airflow:
  build: ./airflow
  command: >
    bash -c "
      airflow db init && 
      airflow users create --username admin --password admin --firstname Admin --lastname User --role Admin --email admin@example.com &&
      airflow scheduler &
      airflow webserver
    "
  # 其余配置保持不变

用&把scheduler放到后台，再启动webserver，这样两个进程会同时运行，无需手动执行。

内容的提问来源于stack exchange，提问作者Chris