首次基于Airflow+Docker搭建ML项目：Flask用户创建失败及404报错求助

首次在IntelliJ中用Airflow和Docker搭建机器学习项目，遇到Flask用户未创建的警告，同时Airflow UI弹出404错误。以下是相关配置代码及报错日志：

Dockerfile代码 #

FROM puckel/docker-airflow:2.1.0
FROM apache/airflow:2.1.0
COPY requirement.txt .
RUN pip install -r requirement.txt

docker-compose.yml代码 #

version: '3.7'

services:
  postgres:
    image: postgres:9.6
    environment:
      - POSTGRES_USER=airflow
      - POSTGRES_PASSWORD=airflow
      - POSTGRES_DB=airflow
    logging:
      options:
        max-size: 10m
        max-file: "3"

  webserver:
    build: ./dockerfiles
    restart: always
    depends_on:
      - postgres
    environment:
      - LOAD_EX=n
      - EXECUTOR=Local
    logging:
      options:
        max-size: 10m
        max-file: "3"
    volumes:
      - ./dags:/usr/local/airflow/dags
      - ./plugins:/usr/local/airflow/plugins

    ports:
      - "8081:8080"
    command: webserver
    healthcheck:
      test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
      interval: 30s
      timeout: 30s
      retries: 3

报错日志 #

project_p-webserver-1  | [2023-08-11 11:57:30,674] {manager.py:784} WARNING - No user yet created, use flask fab command to do it.
project_p-webserver-1  | [2023-08-11 11:57:30,677] {manager.py:784} WARNING - No user yet created, use flask fab command to do it.
project_p-webserver-1  | [2023-08-11 11:57:30,796] {manager.py:784} WARNING - No user yet created, use flask fab command to do it.
project_p-webserver-1  | [2023-08-11 11:57:30,804] {manager.py:784} WARNING - No user yet created, use flask fab command to do it.

1. 修复Dockerfile的多FROM错误 #

Dockerfile中连续两个FROM属于无效用法，第二个FROM apache/airflow:2.1.0会完全覆盖第一个镜像内容，直接删除第一个FROM即可：

FROM apache/airflow:2.1.0
COPY requirement.txt .
RUN pip install -r requirement.txt

2. 补充Airflow数据库连接环境变量 #

Airflow 2.x需要明确配置数据库连接信息，在webserver服务的environment中添加：

environment:
  - LOAD_EX=n
  - EXECUTOR=Local
  - AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres/airflow
  - AIRFLOW__WEBSERVER__EXPOSE_CONFIG=True

3. 修正Airflow文件路径 #

apache/airflow官方镜像的默认工作目录是/opt/airflow，而非/usr/local/airflow，需修改挂载路径和健康检查路径：

volumes:
  - ./dags:/opt/airflow/dags
  - ./plugins:/opt/airflow/plugins
  - ./logs:/opt/airflow/logs  # 新增日志目录挂载，避免权限问题

healthcheck:
  test: ["CMD-SHELL", "[ -f /opt/airflow/airflow-webserver.pid ]"]

4. 添加数据库初始化和用户创建逻辑 #

Airflow 2.x需要手动初始化数据库并创建用户，可通过两种方式实现：

方式一：新增初始化服务（推荐生产环境） #

在docker-compose.yml中添加init服务，负责一次性完成初始化：

services:
  # ... 保留原有postgres、webserver服务 ...
  init:
    build: ./dockerfiles
    depends_on:
      postgres:
        condition: service_healthy
    environment:
      - AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres/airflow
    command: >
      bash -c "airflow db init &&
               airflow users create \
                 --username admin \
                 --firstname Admin \
                 --lastname User \
                 --role Admin \
                 --email admin@example.com \
                 --password admin"

启动顺序：先执行docker-compose run init，再运行docker-compose up -d。

方式二：修改webserver启动命令（适合测试环境） #

将webserver的command改为先初始化再启动webserver：

command: >
  bash -c "airflow db init &&
           airflow users create \
             --username admin \
             --firstname Admin \
             --lastname User \
             --role Admin \
             --email admin@example.com \
             --password admin &&
           airflow webserver"

5. 确保Postgres服务就绪 #

depends_on仅保证启动顺序，需添加健康检查确保Postgres就绪：

postgres:
  image: postgres:9.6
  environment:
    - POSTGRES_USER=airflow
    - POSTGRES_PASSWORD=airflow
    - POSTGRES_DB=airflow
  logging:
    options:
      max-size: 10m
      max-file: "3"
  healthcheck:
    test: ["CMD-SHELL", "pg_isready -U airflow"]
    interval: 10s
    timeout: 5s
    retries: 5

同时修改webserver和init服务的depends_on：

depends_on:
  postgres:
    condition: service_healthy

内容的提问来源于stack exchange，提问作者Ashwini Choudhary