Airflow KubernetesPodOperator连接EKS/ECR失败，请求技术支持

已执行操作 #

• 创建ECR镜像拉取密钥：

kubectl create secret docker-registry ecr --docker-server=xxxx.dkr.ecr.$region.amazonaws.com --docker-username=AWS --docker-password=$(shell aws ecr get-login-password --region $region)

DAG代码 #

import os
from airflow import DAG
from airflow.kubernetes.secret import Secret
from datetime import datetime, timedelta
from kubernetes.client import models as k8s
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import (
    KubernetesPodOperator,
)

with DAG(
    dag_id="eks_poc",....
) as dag:
    eks_k8s = KubernetesPodOperator(
        namespace='eks-poc-namespace',
        cluster_context="arn:aws:eks:$region:xxxx:cluster/eks-poc",
        image='xxxx.dkr.ecr.$region.amazonaws.com/python-app',
        image_pull_secrets=[k8s.V1LocalObjectReference('ecr')],
        cmds=["python3", "/opt/app/app.py"],
        labels={"FOO": "Val"},
        name="eks-poc-pod",
        is_delete_operator_pod=True,
        in_cluster=False,
        task_id="task-two",
        get_logs=True,
        do_xcom_push=True
    )

报错信息 #

[2022-09-14 05:03:40,124] {refresh_config.py:71} ERROR - [Errno 2] No such file or directory: 'aws': 'aws'
[2022-09-14 05:03:40,758] {taskinstance.py:1482} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1138, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1341, in _execute_task
    result = task_copy.execute(context=context)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 350, in execute
    pod_list = client.list_namespaced_pod(self.namespace, label_selector=label_selector)
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 12803, in list_namespaced_pod
    (data) = self.list_namespaced_pod_with_http_info(namespace, **kwargs)  # noqa: E501
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 12905, in list_namespaced_pod_with_http_info
    collection_formats=collection_formats)
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 345, in call_api
    _preload_content, _request_timeout)
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 176, in __call_api
    _request_timeout=_request_timeout)
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 366, in request
    headers=headers)
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 241, in GET
    query_params=query_params)
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 231, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '1f2b7b45-4d1d-4178-9c8c-8709c16fbe32', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '0fb27af0-e840-4042-9fc4-f4d4fe05c643', 'X-Kubernetes-Pf-Prioritylevel-Uid': '89864181-64fb-47b2-8394-fcb2f0cbb89e', 'Date': 'Wed, 14 Sep 2022 05:03:40 GMT', 'Content-Length': '279'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group \"\" in the namespace \"eks-poc-namespace\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}

关键问题分析 #

1. 前置报错No such file or directory: 'aws': 'aws'，说明Airflow运行环境缺少AWS CLI工具，而EKS的kubeconfig认证依赖AWS CLI刷新访问token
2. 后续403错误显示以system:anonymous身份访问EKS，证明Airflow未正确加载kubeconfig中的认证信息，导致身份验证失败

解决方案建议 #

1. 给Airflow运行环境安装AWS CLI #

• 若Airflow运行在容器中，修改镜像Dockerfile添加安装步骤：

  # Python基础镜像示例
  RUN pip install awscli
  # Debian/Ubuntu基础镜像示例
  RUN apt-get update && apt-get install -y awscli
  

• 本地部署的Airflow，直接在运行Airflow的机器上安装AWS CLI

2. 确保Airflow能访问正确的kubeconfig #

• 确认Airflow运行用户（如airflow用户）的家目录下存在.kube/config，且权限为可读
• 若kubeconfig不在默认路径，可通过环境变量指定：

  export KUBECONFIG=/path/to/your/kubeconfig
  

  或在DAG代码中通过参数指定：

  eks_k8s = KubernetesPodOperator(
      # 其他参数...
      kube_config="/path/to/your/kubeconfig",
      in_cluster=False,
  )
  

3. 验证kubeconfig有效性 #

以Airflow运行用户身份执行命令，确认能正常访问EKS：

su - airflow
kubectl --context arn:aws:eks:$region:xxxx:cluster/eks-poc get pods -n eks-poc-namespace

若命令失败，检查kubeconfig的context配置、AWS凭证（~/.aws/credentials或环境变量）是否有效，以及对应IAM实体的EKS访问权限

4. 配置IAM权限 #

确保kubeconfig使用的IAM实体（用户/角色）拥有以下权限：

• eks:DescribeCluster
• 目标namespace的pod列表、创建权限
  可先附加AmazonEKSClusterPolicy托管策略测试，后续再细化权限

5. 确认imagePullSecret的正确性 #

虽然当前报错为认证问题，但需确保ecr密钥存在于目标namespace：

kubectl get secret ecr -n eks-poc-namespace

若不存在，重新创建并指定namespace：

kubectl create secret docker-registry ecr --namespace=eks-poc-namespace --docker-server=xxxx.dkr.ecr.$region.amazonaws.com --docker-username=AWS --docker-password=$(aws ecr get-login-password --region $region)

内容的提问来源于stack exchange，提问作者Lee