Apache Airflow单个DAG报FileNotFoundError错误排查求助

Let’s walk through why only this specific DAG is throwing the FileNotFoundError: [Errno 2] No such file or directory: /home/airflow/composer_kube_config error, and how to fix it.

First, let’s recap the key context from the Cloud Composer docs you shared:

Only name, namespace, image, and task_id are required to create a KubernetesPodOperator. In Cloud Composer, currently the operator defaults to using the config file found at /home/airflow/composer_kube_config if no config_file parameter is specified. By default it will contain the credentials for Cloud Composer's Google Kubernetes Engine cluster that is created upon environment creation.

Since other DAGs using KubernetesPodOperator work fine, the issue is isolated to how this specific DAG is configured. Here are the most likely causes and fixes:

1. Missing Default Arguments That Include Kube Config Settings #

Looking at your DAG code, you import DEFAULT_ARGS from utils.constants but don’t pass it to your DAG definition. It’s highly probable that your working DAGs use default_args=DEFAULT_ARGS, and this config includes either an explicit config_file path for KubernetesPodOperator or other authentication settings that ensure the kube config is accessible.

Fix: #

Add the default_args parameter to your DAG definition:

with DAG(
    'ftp_file_poller',
    schedule_interval="55 6 * * *",
    start_date=datetime.datetime(2020,7,1),
    default_args=DEFAULT_ARGS  # Add this line
) as dag:

2. Explicitly Specify the config_file Path #

If DEFAULT_ARGS doesn’t include kube config settings, you can directly set the config_file parameter in your KubernetesPodOperator to ensure it uses the correct path. This overrides the default lookup and removes ambiguity.

Fix: #

Update your KubernetesPodOperator initialization:

poller = KubernetesPodOperator(
    secrets=[CLOUD_COMPOSER_SERVICE_ACCOUNT_SECRET],
    task_id='ftp-file-poller',
    name='ftp-polling',
    cmds=['ftp-poller'],
    namespace='default',
    image='us.gcr.io/<our gcp project>/ftp-poller:v7',
    is_delete_operator_pod=True,
    get_logs=True,
    volumes=[volume],
    volume_mounts=[volume_mount],
    config_file='/home/airflow/composer_kube_config'  # Explicitly set the config path
)

3. Verify Volume/Secret Configurations Aren’t Interfering #

While less likely, your custom Volume and VolumeMount could be accidentally modifying the worker’s file system or permissions, preventing access to the kube config file. To test this:

Fix: #

Temporarily comment out the volumes and volume_mounts parameters in your KubernetesPodOperator and run the DAG. If the error disappears, you’ll need to adjust your volume configuration to avoid conflicting with the worker’s required files.

4. Check Worker Permissions (Last Resort) #

Since other DAGs work, this is unlikely, but you can confirm that the Airflow worker process running your DAG has read access to /home/airflow/composer_kube_config. You can do this by running a simple test DAG that executes ls /home/airflow/ to verify the file exists.

内容的提问来源于stack exchange，提问作者Nathan McKaskle