Airflow特定DAG停止执行求助：调度器正常但任务中断（v1.8.0）

Hey there, let's break down why your site access data DAG stopped running while all other tasks work reliably. Here's a step-by-step guide to track down the issue:

1. Start with Logs – Your First Clue #

Logs are the fastest way to pinpoint problems, so let's check two critical locations:

• Scheduler Logs: Navigate to your Airflow log directory (usually ~/airflow/logs/scheduler or /var/log/airflow/scheduler) and run tail -f scheduler.log to watch real-time activity. Look for error messages or warnings tied to your DAG ID – this will tell you if the scheduler is even attempting to trigger the DAG, or if there's a scheduling block.
• Task Instance Logs: Each task in your DAG has a dedicated log folder at ~/airflow/logs/<your-dag-id>/<task-id>/<execution-date>. Open the log for the most recent execution date that should have run. Look for stack traces, connection timeouts, or permission errors – this will reveal if the task failed mid-execution.

2. Verify DAG Configuration & Status #

Sometimes the simplest fixes are the easiest to overlook:

• Check if the DAG is paused in the Airflow UI – look for the toggle switch next to your DAG name; if it's gray, click it to resume scheduling.
• Double-check the schedule_interval in your DAG code. A typo in the cron expression (e.g., 0 0 * * 7 instead of 0 0 * * * for daily runs) or accidentally setting it to @never will stop all triggers.
• Confirm the start_date and end_date values. If end_date is set to a past date, the DAG will stop scheduling new runs entirely.

3. Inspect Task Dependencies & Manual Trigger Test #

• Use the Graph View in the Airflow UI for your DAG. Are any upstream tasks marked red (failed)? If an upstream task is stuck in a failed state, downstream tasks won't run until it's resolved.
• Try manually triggering the DAG via the UI (click the "Trigger DAG" button). If it runs successfully manually, the issue is likely with scheduling logic (like a schedule_interval misconfiguration or time zone mismatch). If it fails manually, the problem lies in the task code or its dependencies.
• Audit your task code: Did the site access data source change? For example, did permissions on the S3 bucket storing logs get updated? Did the API key for fetching access data expire? Are there hardcoded file paths that no longer exist on the EC2 instance?

4. Check System Resources & LocalExecutor Health #

Since you're using LocalExecutor, it relies on your EC2 instance's resources and the scheduler process:

• Run htop or top to check CPU and memory usage. If your instance is maxed out (high CPU, low free memory), the scheduler might not be able to spawn new task processes, or tasks could get killed by the OOM (Out of Memory) killer.
• Verify the scheduler is actually running: Run ps aux | grep airflow scheduler. If you don't see the process, restart it with pkill airflow && airflow scheduler -D.
• Check file descriptor limits: LocalExecutor uses multiple processes, which can hit default file descriptor limits. Run ulimit -n – if it's set to 1024, temporarily increase it with ulimit -n 4096 and test if the DAG runs. To make this permanent, add entries to /etc/security/limits.conf.

5. Audit Airflow Metadata Database #

Airflow stores all scheduling state in its metadata database (you should be using PostgreSQL/MySQL with LocalExecutor, since SQLite doesn't support parallel execution):

• Run airflow checkdb (the correct command for Airflow 1.8.0) to confirm the database connection is healthy.
• Connect to your metadata database and check the dag_run table for recent runs of your DAG – are there entries with state set to failed or queued that aren't progressing?
• Check the task_instance table: If a task is stuck in the running state but isn't actually executing, it can block future runs. You can manually update its state to failed or success in the database, or use the Airflow UI's "Clear" button to reset the task instance.

内容的提问来源于stack exchange，提问作者aaron