Airflow任务在Kubernetes中疑似重复运行：如何确认与排查？

我当前使用Apache Airflow搭配Kubernetes Executor，发现疑似任务在Kubernetes中重复运行的异常现象，希望确认该情况是否属实并排查原因。

现象观察 #

查看Pod时发现以下情况：

➜  ~ kubectl get pods -n scheduler
NAME                                                                READY   STATUS    RESTARTS   AGE
da-job-boards-pipeline-jobs-normalisation-task-job-title-492abcde   1/1     Running   0          29h
da-job-boards-pipeline-jobs-normalisation-task-locations-tkaabcde   1/1     Running   0          28h
scheduler-scheduler-58f557f548-abcde                                2/2     Running   0          6d14h
scheduler-statsd-7dd4494d4f-abcde                                   1/1     Running   0          13d
scheduler-triggerer-0                                               2/2     Running   0          8d
scheduler-webserver-5546b8dd66-abcde                                1/1     Running   0          3d6h
➜  ~ kubectl get pods -n airflow
NAME                                    READY   STATUS    RESTARTS   AGE
jobs-normalisation-job-title-2bpsbabc   1/1     Running   0          29h
jobs-normalisation-locations-nv670abc   1/1     Running   0          28h

同时，以下两组Pod的日志完全一致：

• jobs-normalisation-job-title-2bpsbabc（airflow命名空间） 和 da-job-boards-pipeline-jobs-normalisation-task-job-title-492abcde（scheduler命名空间）
• jobs-normalisation-locations-nv670abc（airflow命名空间） 和 da-job-boards-pipeline-jobs-normalisation-task-locations-tkaabcde（scheduler命名空间）

配置信息 #

airflow.cfg 相关配置 #

[kubernetes]
airflow_configmap = scheduler-airflow-config
airflow_local_settings_configmap = scheduler-airflow-config
multi_namespace_mode = True
namespace = scheduler
pod_template_file = /opt/airflow/pod_templates/pod_template_file.yaml
worker_container_repository = SECRET.dkr.ecr.eu-west-1.amazonaws.com/airflow
worker_container_tag = efeb_THIS_IS_A_TAG

[kubernetes_executor]
multi_namespace_mode = True
namespace = scheduler
pod_template_file = /opt/airflow/pod_templates/pod_template_file.yaml
worker_container_repository = SECRET.dkr.ecr.eu-west-1.amazonaws.com/airflow
worker_container_tag = efeb_THIS_IS_A_TAG

[logging]
colored_console_log = False
delete_worker_pods = False
encrypt_s3_logs = True
logging_level = INFO
remote_base_log_folder = s3://scheduler-SECRET-eu-west-1/airflow/logs
remote_log_conn_id = aws_conn
remote_logging = True

DAG 中 KubernetesPodOperator 参数 #

'node_selector': {"abcd.com/tenant": "scheduler"},
'tolerations': [k8s.V1Toleration(key="abcd.com/tenant", operator="Equal", value="scheduler")],
'namespace': "airflow",
'service_account_name': "airflow",

疑问 #

1. 这些配置是否会导致Kubernetes中任务重复运行？
2. 如何确认任务确实存在重复运行的情况？
3. 是否有其他人遇到过类似问题？

内容的提问来源于stack exchange，提问作者The Dan