通过Helm Chart部署Airflow时Pod卡在wait-for-airflow-migration问题排查

部署背景 #

• 基于Airflow 2.7稳定版的Helm Chart进行部署
• 计划使用AWS RDS Postgres作为Airflow元数据的外部持久化存储
• 已在values.yaml中配置元数据数据库连接信息：

# Otherwise pass connection values in
metadataConnection:
  user: <db-user>
  pass: <db-password>
  protocol: postgresql
  host: <db-host-name>
  port: 5432
  db: postgres
  sslmode: disable

遇到的报错情况 #

情况1：默认配置下Pod崩溃 #

所有Airflow Pod因数据库迁移超时无响应而崩溃，报错日志如下：

[2024-03-12T07:15:21.124+0000] {db.py:798} INFO - Waiting for migrations... 59 second(s)
Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 59, in main
    args.func(args)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_config.py", line 49, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/db_command.py", line 186, in check_migrations
    db.check_migrations(timeout=args.migration_wait_timeout)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/db.py", line 799, in check_migrations
    raise TimeoutError(TimeoutError: There are still unapplied migrations after 60 seconds. MigrationHead(s) in DB: set() | Migration Head(s) in Source Code: {'405de8318b3a'}

情况2：禁用waitForMigrations后报错 #

修改values.yaml禁用waitForMigrations后，出现数据库未初始化的错误：

[2024-03-12T07:41:20.611+0000] {configuration.py:2066} INFO - Creating new FAB webserver config file in: /opt/airflow/webserver_config.py
ERROR: You need to initialize the database. Please run `airflow db init`. Make sure the command is run using Airflow version 2.7.1.

疑问 #

1. 数据库迁移的作用是什么？使用持久化RDS数据库时是否必须执行？
2. 应该在哪里以及如何执行airflow db init命令？
3. 还可以调整values.yaml中的哪些配置来成功启动Airflow？

内容的提问来源于stack exchange，提问作者Scorpioooooon21