Apache Airflow设置catchup=False未生效问题求助

Hey there, let's break down and fix your two Airflow issues one by one:

This error pops up because your Airflow metadata database wasn't properly initialized—especially common when running in Docker. The job table is part of Airflow's core metadata schema, and its absence means the database setup process wasn't completed.

Fix Steps: #

• First, jump into your running Airflow container:

  docker exec -it <your-airflow-container-name/id> bash
  

• Run the database initialization command (for Airflow 2.x+):

  airflow db init
  

• If you're upgrading from an older Airflow version, also run this to ensure schema compatibility:

  airflow db upgrade
  

• Pro Tip: SQLite is only meant for local testing. For any real-world use, switch to PostgreSQL or MySQL—SQLite has strict concurrency limitations that often trigger odd issues like this, especially with backfill jobs.

Let's clarify how Airflow's catchup logic actually works:

catchup=False stops the scheduler from automatically generating and scheduling task instances between your start_date and the current date. However, if you started the DAG before enabling catchup=False, the scheduler likely already created all those historical task instances. Those pre-existing instances will still run even after you toggle catchup=False.

Fix Steps: #

1. Pause the DAG first in the Airflow Web UI to halt any ongoing runs.
2. Clear all historical task instances for tutorial2:
   • Use the command line:

     airflow tasks clear tutorial2 -s '2015-06-01' -e $(date +%Y-%m-%d)
     

   • Or via Web UI: Go to the DAG's Tree View, select all past task instances, click the "Clear" button, and confirm the action.
3. Verify your DAG configuration:
   • Your code correctly sets 'catchup': False in default_args, and the DAG instance inherits this setting (no override in your code, so this is good).
4. Restart your Airflow scheduler and webserver containers to ensure all config changes take full effect.
5. Check the global config: Confirm the catchup_by_default = False setting in airflow.cfg is being picked up by the container—sometimes mounted config files don't sync properly, so you can verify by running cat airflow.cfg inside the container.

Key Note: #

The Airflow Web UI will still display the timeline starting from your start_date (2015-06-01) even with catchup=False—this is just the visual representation of your DAG's defined time range. What matters is that no new historical task instances are created or run after you clear the old ones and enable catchup=False.

内容的提问来源于stack exchange，提问作者dhalfageme