You need to enable JavaScript to run this app. 本地Kubernetes集群部署Airflow的镜像构建与配置咨询
优惠活动
大模型
产品
解决方案
定价
更多

本地Kubernetes集群部署Airflow的镜像构建与配置咨询

本地K8s环境部署Airflow实操指南

1. 仅包含webserver与scheduler组件的Airflow镜像构建

基于官方Airflow基础镜像裁剪,移除不需要的组件启动逻辑即可,参考Dockerfile如下:

# 可根据需求调整Airflow和Python版本
FROM apache/airflow:2.8.0-python3.10

USER root
# 安装系统依赖
RUN apt-get update && apt-get install -y --no-install-recommends \
    gcc python3-dev \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

USER airflow
# 安装Kubernetes Executor必要依赖
RUN pip install --no-cache-dir apache-airflow-providers-cncf-kubernetes==7.11.0 pymysql

# 移除worker、triggerer、flower等非必要组件的默认启动脚本
RUN rm -f /entrypoint.d/10-airflow-worker /entrypoint.d/10-airflow-triggerer /entrypoint.d/10-airflow-flower

构建命令:docker build -t my-airflow-web-scheduler:2.8.0 .

2. 独立MySQL镜像构建

基于官方MySQL 8.0镜像定制,提前初始化Airflow所需的库和权限,参考配置如下:

  • 同级目录先准备3个文件:Dockerfile、my.cnf、init-airflow-db.sql

Dockerfile内容:

FROM mysql:8.0
# 导入自定义配置
COPY my.cnf /etc/mysql/conf.d/my.cnf
# 导入库初始化脚本,容器首次启动会自动执行
COPY init-airflow-db.sql /docker-entrypoint-initdb.d/

my.cnf内容:

[mysqld]
default-authentication-plugin=mysql_native_password
character-set-server=utf8mb4
collation-server=utf8mb4_unicode_ci
max_connections=200
innodb_buffer_pool_size=1G

init-airflow-db.sql内容:

CREATE DATABASE IF NOT EXISTS airflow CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER IF NOT EXISTS 'airflow'@'%' IDENTIFIED BY 'airflow123';
GRANT ALL PRIVILEGES ON airflow.* TO 'airflow'@'%';
FLUSH PRIVILEGES;

构建命令:docker build -t my-airflow-mysql:8.0 .

3. 使用Kubernetes Executor的airflow.cfg核心配置

仅列出需要修改的核心配置段,其余保持默认即可:

[core]
# 核心配置:指定执行器为KubernetesExecutor
executor = KubernetesExecutor
load_examples = False
dags_folder = /opt/airflow/dags
base_log_folder = /opt/airflow/logs
# 替换为你的K8s中MySQL服务的连接地址
sql_alchemy_conn = mysql+pymysql://airflow:airflow123@airflow-mysql-svc:3306/airflow

[kubernetes]
# 提前在K8s中创建的Airflow专属命名空间
namespace = airflow-namespace
# Worker镜像版本需和前面构建的web/scheduler镜像版本保持一致
worker_container_repository = apache/airflow
worker_container_tag = 2.8.0-python3.10
worker_container_image_pull_policy = IfNotPresent
# Worker资源配置,可根据本地资源调整
worker_requests_cpu = 0.25
worker_requests_memory = 512Mi
worker_limits_cpu = 1
worker_limits_memory = 1Gi
# DAG存储配置,本地测试可绑定持久卷或者提前把DAG打包进Worker镜像
dags_volume_claim = airflow-dags-pvc
# 任务执行完成后自动销毁Worker Pod
delete_worker_pods = True
worker_pods_labels = { "app": "airflow-worker" }

[webserver]
rbac = True
web_server_host = 0.0.0.0
web_server_port = 8080
enable_proxy_fix = True

[scheduler]
standalone_dag_processor = True
scheduler_heartbeat_sec = 5

注意:如果是本地测试没有持久卷,可把dags_volume_claim配置注释,改为将DAG文件提前打包到Worker镜像中。

4. 本地部署参考步骤

  • 本地K8s如果是minikube/kind环境,先把构建好的镜像导入集群:minikube image load my-airflow-web-scheduler:2.8.0 my-airflow-mysql:8.0
  • 提前创建命名空间:kubectl create namespace airflow-namespace
  • 先部署MySQL服务,参考Deployment示例:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: airflow-mysql
  namespace: airflow-namespace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: airflow-mysql
  template:
    metadata:
      labels:
        app: airflow-mysql
    spec:
      containers:
      - name: mysql
        image: my-airflow-mysql:8.0
        ports:
        - containerPort: 3306
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: root123
        volumeMounts:
        - name: mysql-data
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-data
        persistentVolumeClaim:
          claimName: airflow-mysql-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: airflow-mysql-svc
  namespace: airflow-namespace
spec:
  selector:
    app: airflow-mysql
  ports:
  - port: 3306
    targetPort: 3306
  type: ClusterIP
  • 把编写好的airflow.cfg做成ConfigMap挂载到scheduler和webserver的容器/opt/airflow/airflow.cfg路径,部署后先执行一次数据库初始化:airflow db init
  • 最后创建管理员用户:airflow users create --username admin --firstname Admin --lastname User --role Admin --email admin@example.com,按提示设置密码即可访问web端。

内容的提问来源于stack exchange,提问作者Mayur

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级,最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分), 限时 9.9 元起

最近更新时间:2026.10.19 03:45:08