Airflow服务崩溃并持续重启问题排查求助

问题背景 #

Airflow服务原本运行正常，运行一段时间后崩溃并进入持续重启循环。已排查系统资源无异常，重启机器无法解决问题，怀疑故障由Redis或PostgreSQL引发。

关键日志片段 #

repository-redis-1 exited with code 0
repository-redis-1 has been recreated
repository-postgres-1 exited with code 0
repository-postgres-1 has been recreated
repository-postgres-1 exited with code 0
repository-postgres-1 has been recreated
repository-airflow-triggerer-1 exited with code 0
repository-airflow-triggerer-1 has been recreated
repository-airflow-webserver-1 exited with code 0
repository-airflow-webserver-1 has been recreated
repository-airflow-triggerer-1 exited with code 143
repository-airflow-triggerer-1 has been recreated
repository-airflow-webserver-1 exited with code 143
repository-airflow-webserver-1 has been recreated
repository-postgres-1           |
repository-postgres-1           | PostgreSQL Database directory appears to contain a database; Skipping initialization
repository-postgres-1           |
repository-postgres-1           | 2024-04-08 12:47:18.867 UTC [1] LOG:  starting PostgreSQL 13.14 (Debian 13.14-1.pgdg120+2) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
repository-postgres-1           | 2024-04-08 12:47:18.868 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
repository-postgres-1           | 2024-04-08 12:47:18.868 UTC [1] LOG:  listening on IPv6 address "::", port 5432
repository-postgres-1           | 2024-04-08 12:47:18.875 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
repository-postgres-1           | 2024-04-08 12:47:18.885 UTC [27] LOG:  database system was shut down at 2024-04-08 12:47:14 UTC
repository-postgres-1           | 2024-04-08 12:47:18.903 UTC [1] LOG:  database system is ready to accept connections
repository-airflow-init-1       | The container is run as root user. For security, consider using a regular user account.
repository-airflow-init-1       |
repository-airflow-init-1       | DB: postgresql+psycopg2://airflow:***@postgres/airflow
repository-airflow-init-1       | Performing upgrade to the metadata database postgresql+psycopg2://airflow:***@postgres/airflow
repository-airflow-init-1       | INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
repository-airflow-init-1       | INFO  [alembic.runtime.migration] Will assume transactional DDL.
repository-airflow-init-1       | Database migrating done!
repository-airflow-init-1       | /home/airflow/.local/lib/python3.11/site-packages/flask_limiter/extension.py:336 UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
repository-airflow-init-1       | airflow already exist in the db
repository-airflow-init-1       | 2.7.1
repository-airflow-init-1 exited with code 0
repository-redis-1              | 1:signal-handler (1712580463) Received SIGTERM scheduling shutdown...
repository-redis-1              | 1:M 08 Apr 2024 12:47:43.050 * User requested shutdown...
repository-redis-1              | 1:M 08 Apr 2024 12:47:43.050 * Saving the final RDB snapshot before exiting.
repository-postgres-1           | 2024-04-08 12:47:43.044 UTC [1] LOG:  received fast shutdown request
repository-postgres-1           | 2024-04-08 12:47:43.053 UTC [1] LOG:  aborting any active transactions
repository-postgres-1           | 2024-04-08 12:47:43.056 UTC [1] LOG:  background worker "logical replication launcher" (PID 33) exited with exit code 1
repository-postgres-1           | 2024-04-08 12:47:43.059 UTC [28] LOG:  shutting down
repository-redis-1              | 1:M 08 Apr 2024 12:47:43.066 * DB saved on disk
repository-redis-1              | 1:M 08 Apr 2024 12:47:43.066 # Redis is now ready to exit, bye bye...
repository-postgres-1           | 2024-04-08 12:47:43.138 UTC [1] LOG:  database system is shut down
repository-redis-1 exited with code 0
repository-redis-1 has been recreated
repository-postgres-1 exited with code 0
repository-postgres-1 has been recreated
repository-postgres-1 exited with code 0

排查与修复步骤 #

退出码含义解析 #

• 退出码0：容器正常终止，通常是收到外部停止信号（如SIGTERM）
• 退出码143：容器收到SIGTERM信号后主动终止，一般是Docker或系统触发的关停操作

PostgreSQL排查修复 #

• 检查数据目录权限：确保PostgreSQL挂载的本地目录权限匹配容器内postgres用户（默认UID/GID为999），执行命令修正：

  chown -R 999:999 /path/to/your/postgres/data
  

• 验证数据库完整性：进入PostgreSQL容器执行完整性检测：

  docker exec -it repository-postgres-1 pg_checksdb
  

  若检测到损坏，需使用备份文件恢复数据库
• 查看完整运行日志：获取更多PostgreSQL启动与运行细节，排查连接或配置问题：

  docker logs -f repository-postgres-1
  

Redis排查修复 #

• 检查持久化文件：若使用RDB/AOF持久化，验证文件是否损坏。可先备份旧文件后删除，再重启Redis容器：

  mv /path/to/redis/dump.rdb /path/to/redis/dump.rdb.bak
  docker-compose restart redis
  

• 验证网络连通性：测试Airflow容器与Redis的网络连接：

  docker exec repository-airflow-webserver-1 ping -c 3 redis
  

  同时确认Airflow配置中broker_url的Redis地址无误

Airflow相关修复 #

• 重新初始化元数据：确保元数据结构正确：

  docker-compose run airflow-init
  

• 清理容器与卷：彻底移除所有容器和关联卷（注意提前备份数据）后重新启动：

  docker-compose down -v
  docker-compose up -d
  

• 检查核心配置：确认core/sql_alchemy_conn（PostgreSQL连接）、celery/broker_url（Redis连接）等配置项无错误

内容的提问来源于stack exchange，提问作者Mosy Mosy