Airflow K8s执行器挂载EFS ReadWriteMany卷100Pod时超时
2026-7-13
AWS EFS挂载超时问题：Airflow并行Pod数超25时失败
运行Dags/test_parallelism.py中的DAG时，挂载AWS EFS卷的场景下，Pod数量≤25可正常运行，但增加到100时出现挂载超时错误，报错如下：
Unable to attach or mount volumes: unmounted volumes=[logs], unattached volumes=[logs config backups kube-api-access-jxz9w]: timed out waiting for the condition Unable to attach or mount volumes: unmounted volumes=[logs], unattached volumes=[backups kube-api-access-q6b8x logs config]: timed out waiting for the condition
EFS卷已设置为ReadWriteMany访问模式，理论上支持所有Kubernetes Pod挂载。当前所有Pod挂载两个EFS卷：一个通过DAG的Pod override配置，另一个是Airflow日志卷，日志PVC配置如下：
persistence: # Enable persistent volume for storing logs enabled: true # Volume size for logs size: 14Gi # Annotations for the logs PVC annotations: {} # If using a custom storageClass, pass name here storageClassName: "efs-sc" ## the name of an existing PVC to use existingClaim: "airflow-logs"
以下是挂载失败Pod的kubectl日志信息：
Name: test-1000-task-1-task-44-ff046add566c46bdb78ead1aa72d4e6c Namespace: sb-jniravel Priority: 0 Node: ip-10-0-133-146.ec2.internal/10.0.133.146 Start Time: Wed, 16 Aug 2023 09:21:57 -0500 Labels: airflow-worker=1188 airflow_version=2.6.0 component=worker dag_id=test_1000_task_1 kubernetes_executor=True release=airflow run_id=manual__2023-08-16T142155.7297460000-c3a08be2d task_id=task_44 tier=airflow try_number=1 Annotations: dag_id: test_1000_task_1 openshift.io/scc: airflow-cluster-scc run_id: manual__2023-08-16T14:21:55.729746+00:00 seccomp.security.alpha.kubernetes.io/pod: runtime/default task_id: task_44 try_number: 1 Status: Pending IP: IPs: <none> Containers: base: Container ID: Image: truu.jfrog.io/airflow-etl-repo/airflow:v37 Image ID: Port: <none> Host Port: <none> Args: airflow tasks run test_1000_task_1 task_44 manual__2023-08-16T14:21:55.729746+00:00 --local --subdir DAGS_FOLDER/test_parallelism.py State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment: AIRFLOW__CORE__EXECUTOR: LocalExecutor AIRFLOW__CORE__FERNET_KEY: <set to the key 'fernet-key' in secret 'airflow-fernet-key'> Optional: false AIRFLOW__CORE__SQL_ALCHEMY_CONN: <set to the key 'connection' in secret 'airflow-airflow-metadata'> Optional: false AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: <set to the key 'connection' in secret 'airflow-airflow-metadata'> Optional: false AIRFLOW_CONN_AIRFLOW_DB: <set to the key 'connection' in secret 'airflow-airflow-metadata'> Optional: false AIRFLOW__WEBSERVER__SECRET_KEY: <set to the key 'webserver-secret-key' in secret 'airflow-webserver-secret-key'> Optional: false AIRFLOW__CORE__DEFAULT_POOL_TASK_SLOT_COUNT: 500 AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__CORE__DEFAULT_POOL_TASK_SLOT_COUNT: 500 AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT: 360.0 AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT: 360.0 AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_SIZE: -1 AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_SIZE: -1 AIRFLOW__CORE__PARALLELISM: 500 AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__CORE__PARALLELISM: 500 AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG: 500 AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG: 500 AIRFLOW__SCHEDULER__PARSING_PROCESSES: 32 AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__SCHEDULER__PARSING_PROCESSES: 32 AIRFLOW__SCHEDULER__SCHEDULER_HEALTH_CHECK_THRESHOLD: 60 AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__SCHEDULER__SCHEDULER_HEALTH_CHECK_THRESHOLD: 60 AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG: 500 AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG: 500 AIRFLOW__CORE__DAG_FILE_PROCESSOR_TIMEOUT: 360 AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__CORE__DAG_FILE_PROCESSOR_TIMEOUT: 360 AIRFLOW__KUBERNETES_EXECUTOR__WORKER_PODS_CREATION_BATCH_SIZE: 25 AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__KUBERNETES_EXECUTOR__WORKER_PODS_CREATION_BATCH_SIZE: 25 AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL: 600 AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL: 600 AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: 600 AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: 600 AIRFLOW__SCHEDULER__MAX_DAGRUNS_TO_CREATE_PER_LOOP: 500 AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__SCHEDULER__MAX_DAGRUNS_TO_CREATE_PER_LOOP: 500 AIRFLOW__SCHEDULER__MAX_DAGRUNS_PER_LOOP_TO_SCHEDULE: 500 AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__SCHEDULER__MAX_DAGRUNS_PER_LOOP_TO_SCHEDULE: 500 AIRFLOW__SCHEDULER__SCHEDULER_ZOMBIE_TASK_THRESHOLD: 600 AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__SCHEDULER__SCHEDULER_ZOMBIE_TASK_THRESHOLD: 600 parallel_test_count: 50 AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__parallel_test_count: 50 parallel_test_sleep: 60 AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__parallel_test_sleep: 60 AIRFLOW_IS_K8S_EXECUTOR_POD: True Mounts: /opt/airflow/airflow.cfg from config (ro,path="airflow.cfg") /opt/airflow/backups/ from backups (rw) /opt/airflow/config/airflow_local_settings.py from config (ro,path="airflow_local_settings.py") /opt/airflow/logs from logs (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-k76b5 (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: logs: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: airflow-logs ReadOnly: false config: Type: ConfigMap (a volume populated by a ConfigMap) Name: airflow-airflow-config Optional: false backups: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: airflow-s3-pvc ReadOnly: false kube-api-access-k76b5: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 111s default-scheduler Successfully assigned sb-jniravel/test-1000-task-1-task-44-ff046add566c46bdb78ead1aa72d4e6c to ip-10-0-133-146.ec2.internal Warning FailedMount <invalid> kubelet Unable to attach or mount volumes: unmounted volumes=[logs backups], unattached volumes=[kube-api-access-k76b5 logs config backups]: timed out waiting for the condition
解决方案
1. 延长EFS挂载超时时间
在每个Kubernetes节点上编辑/etc/efs/efs-utils.conf，添加或修改以下参数，将挂载超时从默认30秒延长到60秒：
mount_timeout = 60
修改后重启节点或EFS相关服务，给挂载请求足够的处理时间。
2. 控制Pod创建速率
当前Airflow的AIRFLOW__KUBERNETES_EXECUTOR__WORKER_PODS_CREATION_BATCH_SIZE设为25，短时间内多批次创建会导致EFS挂载请求突增：
- 降低批次大小至10，减少单时间窗口内的挂载请求量
- 调整Airflow调度器参数，增加批次创建的间隔时间，避免瞬间请求过载
3. 升级EFS性能模式
标准模式的EFS并发连接数有限，切换到Max I/O性能模式，该模式专为高并发场景设计，支持更多并发连接和吞吐量。
4. 更新EFS客户端版本
确保节点上的amazon-efs-utils包是最新版本，新版本优化了挂载逻辑和缓存策略，能更好处理大规模并行挂载请求。
5. 检查网络与安全组配置
- 确认EFS挂载目标的安全组允许所有Kubernetes节点访问NFS端口（2049）
- 检查节点到EFS挂载目标的网络延迟，高延迟会增加挂载时间，必要时调整节点区域或EFS挂载目标位置
内容的提问来源于stack exchange，提问作者Jay j
相关产品推荐
相关产品推荐
