Airflow调度器异常退出：pickle数据截断错误原因排查

问题场景 #

运行Airflow DAG时，调度器因异常退出，当前DAG执行成功，但后续DAG未被调度，错误日志如下：

[2023-04-06 12:51:53,853] {scheduler_job.py:753} ERROR - Exception when executing SchedulerJob._run_scheduler_loop
    Traceback (most recent call last):
      File "/home/centos/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 736, in _execute
        self._run_scheduler_loop()
      File "/home/centos/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 824, in _run_scheduler_loop
        num_queued_tis = self._do_scheduling(session)
      File "/home/centos/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 898, in _do_scheduling
        self._start_queued_dagruns(session)
      File "/home/centos/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1052, in _start_queued_dagruns
        DagRun.active_runs_of_dags((dr.dag_id for dr in dag_runs), only_running=True, session=session),
      File "/home/centos/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 68, in wrapper
        return func(*args, **kwargs)
      File "/home/centos/.local/lib/python3.7/site-packages/airflow/models/dagrun.py", line 256, in active_runs_of_dags
        query = query.filter(cls.dag_id.in_(list(set(dag_ids))))
      File "/home/centos/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1052, in <genexpr>
        DagRun.active_runs_of_dags((dr.dag_id for dr in dag_runs), only_running=True, session=session),
      File "/home/centos/.local/lib/python3.7/site-packages/sqlalchemy/engine/result.py", line 376, in iterrows
        for row in self._fetchiter_impl():
      File "/home/centos/.local/lib/python3.7/site-packages/sqlalchemy/orm/loading.py", line 120, in chunks
        fetch = cursor._raw_all_rows()
      File "/home/centos/.local/lib/python3.7/site-packages/sqlalchemy/engine/result.py", line 400, in _raw_all_rows
        return [make_row(row) for row in rows]
      File "/home/centos/.local/lib/python3.7/site-packages/sqlalchemy/engine/result.py", line 400, in <listcomp>
        return [make_row(row) for row in rows]
      File "/home/centos/.local/lib/python3.7/site-packages/sqlalchemy/sql/sqltypes.py", line 1816, in process
        return loads(value)
    _pickle.UnpicklingError: pickle data was truncated
    [2023-04-06 12:51:54,859] {process_utils.py:129} INFO - Sending Signals.SIGTERM to group 27030. PIDs of all processes in the group: [27030]
    [2023-04-06 12:51:54,859] {process_utils.py:80} INFO - Sending the signal Signals.SIGTERM to group 27030
    [2023-04-06 12:51:54,991] {process_utils.py:75} INFO - Process psutil.Process(pid=27030, status='terminated', exitcode=0, started='2023-01-31 20:44:27') (27030) terminated with exit code 0
    [2023-04-06 12:51:54,992] {scheduler_job.py:765} INFO - Exited execute loop

可能的原因 #

• 元数据库数据异常：Airflow元数据库中存储的pickle格式序列化数据（如DAG、DAGRun相关数据）被截断，大概率是数据库写入时出现中断（比如磁盘空间不足、数据库连接异常断开），导致读取时无法正常反序列化。
• 磁盘IO故障：磁盘写入过程中出现IO错误，部分pickle数据未完整写入元数据库，后续读取时触发截断错误。
• 版本兼容问题：当前使用的Airflow版本存在序列化/反序列化bug，或者Python版本不兼容（当前为Python3.7，若DAG代码曾在其他Python版本环境生成序列化数据，可能出现格式不兼容）。
• 调度器资源不足：调度器进程内存不足，处理pickle数据时出现截断；或者进程曾被系统强制终止，残留了未完成的数据库写入操作，导致数据不完整。
• 元数据表损坏：dag_run等DAG运行相关的元数据表出现数据损坏，某条记录的pickle字段内容不完整。

内容的提问来源于stack exchange，提问作者Nandeesh Bijoor