Airflow中execution_date与backfill含义及execution_date执行异常求助

Hey there! Let's break down these two key Airflow concepts to clear up your confusion—they can feel tricky at first, but once you wrap your head around the core ideas, everything clicks into place.

What is execution_date? #

First, let's nail down the core meaning of execution_date:

• It’s a logical date, not the actual time your task runs. Think of it as the "scheduled window" the task is meant to process. For example, a daily DAG with an execution_date of 2015-06-01 is supposed to handle data from that day, even if it actually runs the next morning (like 2015-06-02 00:00:00).

Why your airflow test command showed system time #

The issue with your test run boils down to how airflow test works and how your task is defined:

• The airflow test <dag_id> <task_id> <execution_date> command runs a single task in isolation, bypassing the Airflow scheduler. It passes the specified execution_date to the task's context, but your task won’t use it unless you explicitly reference it via Airflow’s templating system.
• In your case, the print_date task is likely using a simple bash_command='date', which calls your system’s native date command—this will always output your local system time, not the logical execution_date. To get the expected timestamp, you need to use Airflow’s template variable like this:

  bash_command='echo {{ execution_date }}'
  

  Update the task to use this template, re-run airflow test tutorial print_date 2015-06-01, and you’ll see the 2015-06-01 timestamp you expected.

What is Backfill? #

Backfill is Airflow’s tool for retroactively running tasks for past execution_date ranges. Here’s the breakdown:

• It’s used when you create a DAG that should have run for historical dates but didn’t (e.g., you set up a daily DAG today but need to process data from the last 30 days).
• Unlike airflow test, backfill uses the Airflow scheduler to run tasks properly: it respects your DAG’s schedule interval, task dependencies, and will execute each task for every execution_date in your specified range.
• Example command to backfill a DAG from 2015-06-01 to 2015-06-07:

  airflow dags backfill -s 2015-06-01 -e 2015-06-07 tutorial
  

  This will trigger the DAG for each date in that range, with each task using the corresponding execution_date as its logical processing window.

内容的提问来源于stack exchange，提问作者pkgajulapalli