Airflow 2.7.2 UI中部分DAG日志不显示问题求助
Airflow 2.7.2(CeleryExecutor)日志显示异常问题
我在本地Docker环境运行Airflow 2.7.2(Python3.9,使用CeleryExecutor),遇到日志显示异常:仅一个DAG的日志能正常展示,其余所有DAG的日志都无法在UI中显示,无法定位问题原因。
airflow.cfg 配置
[core] executor = CeleryExecutor sql_alchemy_conn = postgresql+psycopg2://airflow:airflow@postgres/airflow dags_are_paused_at_creation = true load_examples = false load_default_connections = false fernet_key = [database] sql_alchemy_conn = postgresql+psycopg2://airflow:airflow@postgres/airflow [celery] result_backend = db+postgresql://airflow:airflow@postgres/airflow broker_url = redis://:@redis:6379/0 [smtp] smtp_host = SMTP_HOST smtp_port = SMTP_PORT smtp_user = SMTP_USER smtp_password = SMTP_PASSWORD smtp_mail_from = SMTP_MAIL_FROM [logging] base_log_folder = /opt/airflow/logs logging_level = INFO # logging_config_class = log_config.LOGGING_CONFIG [scheduler] enable_health_check = true [webserver] base_url = http://localhost:8080 auth_backend = airflow.api.auth.backend.deny_all authenticate = false rbac = false
docker-compose.yml 配置
version: '3.8' x-airflow-common: &airflow-common build: . environment: _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-} volumes: - ${AIRFLOW_PROJ_DIR:-.}/dags:/opt/airflow/dags - ${AIRFLOW_PROJ_DIR:-.}/logs:/opt/airflow/logs - ${AIRFLOW_PROJ_DIR:-.}/config:/opt/airflow/config - ${AIRFLOW_PROJ_DIR:-.}/plugins:/opt/airflow/plugins - ./airflow.cfg:/opt/airflow/airflow.cfg user: "${AIRFLOW_UID:-50000}:0" depends_on: redis: condition: service_healthy postgres: condition: service_healthy services: postgres: image: postgres:13 environment: POSTGRES_USER: airflow POSTGRES_PASSWORD: airflow POSTGRES_DB: airflow volumes: - postgres-db-volume:/var/lib/postgresql/data healthcheck: test: ["CMD", "pg_isready", "-U", "airflow"] interval: 10s retries: 5 start_period: 5s restart: always redis: image: redis:latest expose: - 6379 healthcheck: test: ["CMD", "redis-cli", "ping"] interval: 10s timeout: 30s retries: 50 start_period: 30s restart: always airflow-webserver: <<: *airflow-common command: webserver ports: - "8080:8080" healthcheck: test: ["CMD", "curl", "--fail", "http://localhost:8080/health"] interval: 30s timeout: 10s retries: 5 start_period: 30s restart: always depends_on: <<: *airflow-common-depends-on airflow-init: condition: service_completed_successfully env_file: - .env airflow-scheduler: <<: *airflow-common command: scheduler healthcheck: test: ["CMD", "curl", "--fail", "http://localhost:8974/health"] interval: 30s timeout: 10s retries: 5 start_period: 30s restart: always depends_on: <<: *airflow-common-depends-on airflow-init: condition: service_completed_successfully env_file: - .env airflow-worker: <<: *airflow-common command: celery worker healthcheck: test: - "CMD-SHELL" - 'celery --app airflow.providers.celery.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}" || celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"' interval: 30s timeout: 10s retries: 5 start_period: 30s environment: <<: *airflow-common-env DUMB_INIT_SETSID: "0" restart: always depends_on: <<: *airflow-common-depends-on airflow-init: condition: service_completed_successfully airflow-triggerer: <<: *airflow-common command: triggerer healthcheck: test: ["CMD-SHELL", 'airflow jobs check --job-type TriggererJob --hostname "$${HOSTNAME}"'] interval: 30s timeout: 10s retries: 5 start_period: 30s restart: always depends_on: <<: *airflow-common-depends-on airflow-init: condition: service_completed_successfully airflow-init: << : *airflow-common container_name: airflow_init entrypoint: /bin/bash command: - -c - | airflow db init && airflow users create --role Admin --username airflow --password airflow --email airflow@airflow.com --firstname airflow --lastname airflow restart: on-failure
我注意到Airflow 2.7.2存在相关已知问题,但不解的是为什么有一个DAG的日志能正常显示?(日志异常表现为:UI中点击其余DAG的任务日志时,仅显示加载状态,无法展示实际日志内容)
内容的提问来源于stack exchange,提问作者abhinavkm
相关产品推荐
相关产品推荐

