升级Airflow 1.7至1.9后无法运行DAG的问题求助

Hey there, I’ve dealt with a few Airflow upgrade headaches between these versions, so let’s walk through the most likely fixes for your DAGs not running (and missing logs) post-upgrade:

1. Verify Core Airflow Components Are Running & Updated #

First, make sure every part of Airflow is on version 1.9 and functioning properly:

• Check your Airflow version with: airflow version (ensure it returns 1.9.x everywhere—webserver, scheduler, workers)
• List running Airflow processes: ps aux | grep airflow
  • Confirm the scheduler, webserver, and any workers (if using CeleryExecutor) are active and not crashing repeatedly.

2. Validate DAG Compatibility with Airflow 1.9 #

Airflow 1.9 introduced breaking changes from 1.7 that might break your DAGs silently:

• Check for deprecated APIs: For example, some operator parameters or methods (like DAG constructor arguments) changed. Remember that log.logging_mixin module your initdb failed on earlier? Double-check that none of your DAGs or custom operators are importing this deprecated module—it was moved/renamed in 1.9.
• Test individual DAGs:
  • List all recognized DAGs: airflow list_dags (does your "every 40 minutes" DAG show up here?)
  • Run a test task manually to catch errors: airflow test <your_dag_id> <task_id> 2024-03-28T00:00:00
    • This executes the task locally and prints errors directly to your terminal—perfect for catching import issues or compatibility bugs.

3. Fix Log Configuration & Permissions #

Even if you migrated logs, 1.9’s log setup has key differences from 1.7:

• Open your airflow.cfg and verify these settings:
  • base_log_folder: Ensure this path exists and the user running Airflow has read/write access to it.
  • remote_logging: If you’re using remote logs (S3/GCS), confirm the connection details are correct for 1.9.
  • task_log_reader: Should be set to file.task by default unless you’re using remote logs.
• Fix folder permissions if needed: chown -R <airflow_user>:<airflow_group> /path/to/airflow/logs
• After manual trigger, check the log directory directly (not just the UI) to see if any files are being created—sometimes the UI doesn’t show logs even if they exist due to permission issues.

4. Ensure Database Migration Completed Successfully #

You mentioned an initdb failure, but upgrading requires upgradedb, not initdb:

• Run the upgrade command again to be safe: airflow upgradedb
• Verify your database schema matches Airflow 1.9:
  • Connect to your Airflow database (e.g., psql -U airflow_user airflow_db for Postgres)
  • Check for 1.9-specific tables/columns, like dag_run.state or task_instance.operator—missing these can break task scheduling entirely.

5. Check Scheduler Logs for Clues #

The scheduler log is your best friend for silent failures:

• Tail the latest scheduler log: tail -f /path/to/airflow/logs/scheduler/latest.log
• Look for errors like:
  • Failed to import DAG: Indicates a syntax or compatibility issue in your DAG file.
  • Task not queued: Points to executor configuration problems (e.g., Celery broker not reachable).
  • Permission denied: Log or database access issues blocking task execution.

6. Diagnose Manual Trigger Status #

If your DAG shows as "running" but no progress:

• In the Airflow UI, go to DAG Runs → click your manual run → check the Task Instances tab.
  • Are tasks stuck in queued? This means workers aren’t picking up tasks—verify Celery workers are running and connected to the broker.
  • Are tasks in failed status but no logs? The task likely crashed before logging could start—use the airflow test command from step 2 to reproduce the error locally.

内容的提问来源于stack exchange，提问作者Greg Dougherty