本地Kubernetes集群部署Airflow的镜像构建与配置咨询
本地K8s环境部署Airflow实操指南
1. 仅包含webserver与scheduler组件的Airflow镜像构建
基于官方Airflow基础镜像裁剪,移除不需要的组件启动逻辑即可,参考Dockerfile如下:
# 可根据需求调整Airflow和Python版本 FROM apache/airflow:2.8.0-python3.10 USER root # 安装系统依赖 RUN apt-get update && apt-get install -y --no-install-recommends \ gcc python3-dev \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* USER airflow # 安装Kubernetes Executor必要依赖 RUN pip install --no-cache-dir apache-airflow-providers-cncf-kubernetes==7.11.0 pymysql # 移除worker、triggerer、flower等非必要组件的默认启动脚本 RUN rm -f /entrypoint.d/10-airflow-worker /entrypoint.d/10-airflow-triggerer /entrypoint.d/10-airflow-flower
构建命令:docker build -t my-airflow-web-scheduler:2.8.0 .
2. 独立MySQL镜像构建
基于官方MySQL 8.0镜像定制,提前初始化Airflow所需的库和权限,参考配置如下:
- 同级目录先准备3个文件:Dockerfile、my.cnf、init-airflow-db.sql
Dockerfile内容:
FROM mysql:8.0 # 导入自定义配置 COPY my.cnf /etc/mysql/conf.d/my.cnf # 导入库初始化脚本,容器首次启动会自动执行 COPY init-airflow-db.sql /docker-entrypoint-initdb.d/
my.cnf内容:
[mysqld] default-authentication-plugin=mysql_native_password character-set-server=utf8mb4 collation-server=utf8mb4_unicode_ci max_connections=200 innodb_buffer_pool_size=1G
init-airflow-db.sql内容:
CREATE DATABASE IF NOT EXISTS airflow CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; CREATE USER IF NOT EXISTS 'airflow'@'%' IDENTIFIED BY 'airflow123'; GRANT ALL PRIVILEGES ON airflow.* TO 'airflow'@'%'; FLUSH PRIVILEGES;
构建命令:docker build -t my-airflow-mysql:8.0 .
3. 使用Kubernetes Executor的airflow.cfg核心配置
仅列出需要修改的核心配置段,其余保持默认即可:
[core] # 核心配置:指定执行器为KubernetesExecutor executor = KubernetesExecutor load_examples = False dags_folder = /opt/airflow/dags base_log_folder = /opt/airflow/logs # 替换为你的K8s中MySQL服务的连接地址 sql_alchemy_conn = mysql+pymysql://airflow:airflow123@airflow-mysql-svc:3306/airflow [kubernetes] # 提前在K8s中创建的Airflow专属命名空间 namespace = airflow-namespace # Worker镜像版本需和前面构建的web/scheduler镜像版本保持一致 worker_container_repository = apache/airflow worker_container_tag = 2.8.0-python3.10 worker_container_image_pull_policy = IfNotPresent # Worker资源配置,可根据本地资源调整 worker_requests_cpu = 0.25 worker_requests_memory = 512Mi worker_limits_cpu = 1 worker_limits_memory = 1Gi # DAG存储配置,本地测试可绑定持久卷或者提前把DAG打包进Worker镜像 dags_volume_claim = airflow-dags-pvc # 任务执行完成后自动销毁Worker Pod delete_worker_pods = True worker_pods_labels = { "app": "airflow-worker" } [webserver] rbac = True web_server_host = 0.0.0.0 web_server_port = 8080 enable_proxy_fix = True [scheduler] standalone_dag_processor = True scheduler_heartbeat_sec = 5
注意:如果是本地测试没有持久卷,可把dags_volume_claim配置注释,改为将DAG文件提前打包到Worker镜像中。
4. 本地部署参考步骤
- 本地K8s如果是minikube/kind环境,先把构建好的镜像导入集群:
minikube image load my-airflow-web-scheduler:2.8.0 my-airflow-mysql:8.0 - 提前创建命名空间:
kubectl create namespace airflow-namespace - 先部署MySQL服务,参考Deployment示例:
apiVersion: apps/v1 kind: Deployment metadata: name: airflow-mysql namespace: airflow-namespace spec: replicas: 1 selector: matchLabels: app: airflow-mysql template: metadata: labels: app: airflow-mysql spec: containers: - name: mysql image: my-airflow-mysql:8.0 ports: - containerPort: 3306 env: - name: MYSQL_ROOT_PASSWORD value: root123 volumeMounts: - name: mysql-data mountPath: /var/lib/mysql volumes: - name: mysql-data persistentVolumeClaim: claimName: airflow-mysql-pvc --- apiVersion: v1 kind: Service metadata: name: airflow-mysql-svc namespace: airflow-namespace spec: selector: app: airflow-mysql ports: - port: 3306 targetPort: 3306 type: ClusterIP
- 把编写好的airflow.cfg做成ConfigMap挂载到scheduler和webserver的容器
/opt/airflow/airflow.cfg路径,部署后先执行一次数据库初始化:airflow db init - 最后创建管理员用户:
airflow users create --username admin --firstname Admin --lastname User --role Admin --email admin@example.com,按提示设置密码即可访问web端。
内容的提问来源于stack exchange,提问作者Mayur
相关产品推荐
相关产品推荐

