Airflow-K8s问题：使用KubernetesPodOperator无法挂载HostPath

已部署ADF托管的Airflow实例，尝试调度一个DAG。在DAG中通过KubernetesPodOperator运行位于主机路径/opt/airflow/dags下的shell脚本，该脚本会向K8s集群提交kubectl命令，但Pod始终无法在K8s上启动。

dag.py代码 #

from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
from datetime import datetime, timedelta

from kubernetes.client import models as k8s

# Define your default_args and DAG
default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(1970, 1, 1),
    'retries': 0,
    'retry_delay': timedelta(minutes=5),
}

dag = DAG(
    'test_operations_dag',
    default_args=default_args,
    description='DAG for test Operations',
    #schedule_interval=timedelta(days=1),
    schedule_interval=None,
    concurrency=80,
)

# Correct instantiation using V1HostPathVolumeSource
host_path_volume_source = k8s.V1HostPathVolumeSource(
    path="/opt/airflow/dags",
    type='Directory'  # optional: specify the type of the hostPath
)

volume = k8s.V1Volume(
    name="dags",
    host_path=host_path_volume_source
)

volume_mounts = [
    k8s.V1VolumeMount(
        mount_path="/dags", name="dags"
    )
]

status_task = KubernetesPodOperator(
    task_id=f'status-task-vbuv',
    namespace="spark-apps",
    image="bitnami/kubectl:latest",
    cmds=["sh", "-c"],
    arguments=[
        f"cp /dags/sparkapplication_spark-pi-fixed2.sh /tmp/sparkapplication_spark-pi-fixed2.sh && chmod +x /tmp/sparkapplication_spark-pi-fixed2.sh && /tmp/sparkapplication_spark-pi-fixed2.sh"
    ],
    service_account_name="airflow-sparkapp",
    get_logs=True,
    kubernetes_conn_id="k8s-airflow",
    dag=dag,
    volumes=[volume],
    volume_mounts=volume_mounts,
)      
# Set up dependencies
status_task

DAG日志错误信息 #

[2024-04-18, 05:01:52 UTC] {pod_manager.py:313} WARNING - Pod not yet started: status-task-vbuv-8h4po563
[2024-04-18, 05:01:52 UTC] {pod.py:726} INFO - Deleting pod: status-task-vbuv-8h4po563
[2024-04-18, 05:01:52 UTC] {taskinstance.py:1824} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py", line 551, in execute_sync
    self.await_pod_start(pod=self.pod)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py", line 513, in await_pod_start
    self.pod_manager.await_pod_start(pod=pod, startup_timeout=self.startup_timeout_seconds)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 320, in await_pod_start
    raise PodLaunchFailedException(msg)
airflow.providers.cncf.kubernetes.utils.pod_manager.PodLaunchFailedException

1. 验证HostPath卷的有效性与权限 #

• 确认ADF Airflow Worker节点上/opt/airflow/dags目录存在且为目录（与type='Directory'配置匹配），若路径不存在或为文件，K8s会拒绝启动Pod。
• 检查目录权限：执行ls -ld /opt/airflow/dags，确保权限至少为755，让Pod运行用户拥有访问权限。
• 注意：ADF托管环境可能限制HostPath挂载的使用，这是导致Pod启动失败的常见原因，建议改用PersistentVolumeClaim（PVC）或把脚本打包到镜像中替代HostPath。

2. 检查ServiceAccount权限 #

• 查看airflow-sparkapp在spark-apps命名空间下的权限：

  kubectl describe sa airflow-sparkapp -n spark-apps
  kubectl describe rolebinding -n spark-apps | grep airflow-sparkapp
  

• 确保该ServiceAccount拥有创建Pod、访问Pod日志、使用HostPath卷的权限，若权限不足，创建对应的Role和RoleBinding：

  apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  metadata:
    namespace: spark-apps
    name: airflow-pod-role
  rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["create", "get", "list", "delete"]
  - apiGroups: [""]
    resources: ["volumes"]
    verbs: ["use"]
  ---
  apiVersion: rbac.authorization.k8s.io/v1
  kind: RoleBinding
  metadata:
    namespace: spark-apps
    name: airflow-pod-binding
  subjects:
  - kind: ServiceAccount
    name: airflow-sparkapp
    namespace: spark-apps
  roleRef:
    kind: Role
    name: airflow-pod-role
    apiGroup: rbac.authorization.k8s.io
  

3. 查看Pod启动失败的详细事件 #

DAG运行失败后，立即执行以下命令获取Pod的具体失败原因：

# 查找对应的Pod名称
kubectl get pods -n spark-apps | grep status-task-vbuv
# 查看Pod详细状态与事件
kubectl describe pod <pod-full-name> -n spark-apps

常见失败原因：

• ImagePullBackOff：镜像拉取失败，检查网络连通性或改用国内镜像源。
• FailedMount：HostPath挂载失败，路径不存在或权限不足。
• InvalidVolumeMount：卷配置错误（如卷名称与挂载名称不匹配）。

4. 延长Pod启动超时时间 #

默认启动超时时间可能不足，在KubernetesPodOperator中增加startup_timeout_seconds参数：

status_task = KubernetesPodOperator(
    # 保留原有参数
    startup_timeout_seconds=300,  # 设置为5分钟
)

5. 替代HostPath的更优方案 #

HostPath依赖宿主机环境，在托管Airflow中兼容性差，推荐两种替代方案：

• 将脚本打包到镜像：基于bitnami/kubectl:latest构建自定义镜像，把shell脚本加入镜像中，避免挂载宿主机目录。
• 从存储服务下载脚本：将脚本上传到Azure Blob Storage，在Pod启动时通过az storage blob download或curl下载到容器内执行。

内容的提问来源于stack exchange，提问作者Tad