Airflow HTTP任务长时间无响应持续运行问题排查请求

运行Airflow官方HTTP示例时，出现HTTP任务在UI中长时间运行且无进展的异常，但通过CLI执行该任务却完全正常。

UI执行日志 #

AIRFLOW_CTX_DAG_EMAIL=airflow@example.com
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=example_http_operator
AIRFLOW_CTX_TASK_ID=http_sensor_check
AIRFLOW_CTX_EXECUTION_DATE=2023-02-17T20:53:45.614721+00:00
AIRFLOW_CTX_TRY_NUMBER=1
AIRFLOW_CTX_DAG_RUN_ID=manual__2023-02-17T20:53:45.614721+00:00
[2023-02-17, 20:53:48 UTC] {__init__.py:117} DEBUG - Preparing lineage inlets and outlets
[2023-02-17, 20:53:48 UTC] {__init__.py:155} DEBUG - inlets: [], outlets: []
[2023-02-17, 20:53:48 UTC] {http.py:122} INFO - Poking: 
[2023-02-17, 20:53:48 UTC] {base.py:73} INFO - Using connection ID 'http_default' for task execution.
[2023-02-17, 20:53:48 UTC] {http.py:150} DEBUG - Sending 'GET' to url: https://jsonplaceholder.typicode.com/
[2023-02-17, 20:53:52 UTC] {taskinstance.py:769} DEBUG - Refreshing TaskInstance <TaskInstance: example_http_operator.http_sensor_check manual__2023-02-17T20:53:45.614721+00:00 [running]> from DB
[2023-02-17, 20:53:52 UTC] {base_job.py:240} DEBUG - [heartbeat]
[2023-02-17, 20:53:58 UTC] {taskinstance.py:769} DEBUG - Refreshing TaskInstance <TaskInstance: example_http_operator.http_sensor_check manual__2023-02-17T20:53:45.614721+00:00 [running]> from DB
[2023-02-17, 20:53:58 UTC] {base_job.py:240} DEBUG - [heartbeat]
[2023-02-17, 20:54:03 UTC] {taskinstance.py:769} DEBUG - Refreshing TaskInstance <TaskInstance: example_http_operator.http_sensor_check manual__2023-02-17T20:53:45.614721+00:00 [running]> from DB
[2023-02-17, 20:54:03 UTC] {base_job.py:240} DEBUG - [heartbeat]
[2023-02-17, 20:54:08 UTC] {taskinstance.py:769} DEBUG - Refreshing TaskInstance <TaskInstance: example_http_operator.http_sensor_check manual__2023-02-17T20:53:45.614721+00:00 [running]> from DB
[2023-02-17, 20:54:08 UTC] {base_job.py:240} DEBUG - [heartbeat]
[2023-02-17, 20:54:13 UTC] {taskinstance.py:769} DEBUG - Refreshing TaskInstance <TaskInstance: example_http_operator.http_sensor_check manual__2023-02-17T20:53:45.614721+00:00 [running]> from DB

CLI执行日志 #

AIRFLOW_CTX_DAG_EMAIL=airflow@example.com
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=example_http_operator
AIRFLOW_CTX_TASK_ID=http_sensor_check
AIRFLOW_CTX_EXECUTION_DATE=2023-02-17T21:05:22.781965+00:00
AIRFLOW_CTX_TRY_NUMBER=1
AIRFLOW_CTX_DAG_RUN_ID=__airflow_temporary_run_2023-02-17T21:05:22.781968+00:00__
[2023-02-17 16:05:23,328] {__init__.py:117} DEBUG - Preparing lineage inlets and outlets
[2023-02-17 16:05:23,328] {__init__.py:155} DEBUG - inlets: [], outlets: []
[2023-02-17 16:05:23,329] {http.py:122} INFO - Poking: 
[2023-02-17 16:05:23,332] {base.py:73} INFO - Using connection ID 'http_default' for task execution.
[2023-02-17 16:05:23,332] {http.py:150} DEBUG - Sending 'GET' to url: https://jsonplaceholder.typicode.com/
[2023-02-17 16:05:23,335] {connectionpool.py:1003} DEBUG - Starting new HTTPS connection (1): jsonplaceholder.typicode.com:443
[2023-02-17 16:05:23,667] {connectionpool.py:456} DEBUG - https://jsonplaceholder.typicode.com:443 "GET / HTTP/1.1" 200 None
[2023-02-17 16:05:23,669] {base.py:228} INFO - Success criteria met. Exiting.
[2023-02-17 16:05:23,669] {__init__.py:75} DEBUG - Lineage called with inlets: [], outlets: []
[2023-02-17 16:05:23,670] {taskinstance.py:1329} DEBUG - Clearing next_method and next_kwargs.
[2023-02-17 16:05:23,670] {taskinstance.py:1318} INFO - Marking task as SUCCESS. dag_id=example_http_operator, task_id=http_sensor_check, execution_date=20230217T210522, start_date=, end_date=20230217T210523
[2023-02-17 16:05:23,670] {taskinstance.py:2241} DEBUG - Task Duration set to None
[2023-02-17 16:05:23,696] {cli_action_loggers.py:83} DEBUG - Calling callbacks: []
[2023-02-17 16:05:23,696] {settings.py:407} DEBUG - Disposing DB connection pool (PID 65429)

内容的提问来源于stack exchange，提问作者Bhanu Prakash