Airflow 2.8.0中Celery Worker的broker_url配置位置及默认文件咨询
Airflow 2.8.0 Celery+Redis 配置疑问解答
问题1:broker_url 配置项的放置位置
Airflow 2.x 版本(含2.8.0)已将Celery相关配置整合到主配置文件**airflow.cfg**中,无需去airflow.providers.celery目录寻找单独配置文件——你看到的旧文档内容已过时。
若你的airflow.cfg中没有[celery]配置段,直接手动添加即可,将broker_url置于该段落下:
[celery] broker_url = redis://你的Redis地址:6379/0 # 可补充其他Celery相关配置示例: # result_backend = db+postgresql://airflow_user:密码@localhost:5432/airflow_db # worker_concurrency = 16
问题2:Celery Worker 默认使用的配置文件
Airflow Celery Worker 默认直接读取**airflow.cfg**中的Celery配置段。你启动日志里显示的redis://redis:6379/0是Airflow内置的默认Broker URL值——当未手动配置broker_url时,Worker会自动使用该默认值连接Redis。
你提供的启动日志:
(airflow) my@comp:~/airflow$ airflow celery worker [2023-12-30 11:41:27 +0000] [215202] [INFO] Starting gunicorn 21.2.0 [2023-12-30 11:41:27 +0000] [215202] [INFO] Listening at: http://[::]:8793 (215202) [2023-12-30 11:41:27 +0000] [215202] [INFO] Using worker: sync [2023-12-30 11:41:27 +0000] [215203] [INFO] Booting worker with pid: 215203 [2023-12-30 11:41:27 +0000] [215204] [INFO] Booting worker with pid: 215204 -------------- celery@comp v5.3.6 (emerald-rush) --- ***** ----- -- ******* ---- Linux-5.15.0-91-generic-x86_64-with-glibc2.35 2023-12-30 11:41:27 - *** --- * --- - ** ---------- [config] - ** ---------- .> app: airflow.providers.celery.executors.celery_executor:0x7fd684150b50 - ** ---------- .> transport: redis://redis:6379/0 - ** ---------- .> results: postgresql+psycopg2://airflow_user:**@localhost:5432/airflow_db - *** --- * --- .> concurrency: 16 (prefork) -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker) --- ***** ----- -------------- [queues] .> default exchange=default(direct) key=default
内容的提问来源于stack exchange,提问作者aestet
相关产品推荐
相关产品推荐

