Airflow Docker部署无法找到本地Python脚本及目录(ModuleNotFoundError)
解决Airflow Docker部署中自定义模块导入的ModuleNotFoundError问题
方案一:调整挂载路径与PYTHONPATH配置
这是最规范的解决方式,通过正确挂载项目目录并配置PYTHONPATH让Python识别自定义包:
修改docker-compose.yaml的卷挂载配置
移除原有挂载项目根目录到/opt/airflow的配置,改为挂载项目根目录的上级目录到容器内的指定路径(比如/opt/airflow/project),同时保留dags、logs、plugins的挂载:volumes: - ${AIRFLOW_PROJ_DIR:-.}/dags:/opt/airflow/dags - ${AIRFLOW_PROJ_DIR:-.}/logs:/opt/airflow/logs - ${AIRFLOW_PROJ_DIR:-.}/plugins:/opt/airflow/plugins - ${AIRFLOW_PROJ_DIR:-.}/..:/opt/airflow/project为Airflow服务添加PYTHONPATH环境变量
在airflow-webserver、airflow-scheduler、airflow-worker等核心服务的environment字段中添加PYTHONPATH配置,指向容器内挂载的上级目录:services: airflow-webserver: environment: - PYTHONPATH=/opt/airflow/project # 保留原有其他配置(如image、ports等) airflow-scheduler: environment: - PYTHONPATH=/opt/airflow/project # 保留原有其他配置 airflow-worker: environment: - PYTHONPATH=/opt/airflow/project # 保留原有其他配置
配置完成后,重启Airflow容器,Python会在/opt/airflow/project路径下找到root_package包,from root_package import foo1即可正常导入。
方案二:通过软链接适配现有挂载(不修改导入语句)
如果不想调整挂载路径,可通过在容器内创建软链接的方式让Python识别root_package包:
在docker-compose.yaml的核心服务中添加启动命令,创建软链接并启动服务:
services: airflow-webserver: command: > bash -c "ln -sf /opt/airflow /opt/airflow/root_package && exec airflow webserver" environment: - PYTHONPATH=/opt/airflow # 保留原有其他配置 airflow-scheduler: command: > bash -c "ln -sf /opt/airflow /opt/airflow/root_package && exec airflow scheduler" environment: - PYTHONPATH=/opt/airflow # 保留原有其他配置 airflow-worker: command: > bash -c "ln -sf /opt/airflow /opt/airflow/root_package && exec airflow worker" environment: - PYTHONPATH=/opt/airflow # 保留原有其他配置
该方案通过在容器的/opt/airflow目录下创建名为root_package的软链接指向自身,让Python将当前目录识别为root_package包,从而兼容原有导入语句。
内容的提问来源于stack exchange,提问作者bcsta
相关产品推荐
相关产品推荐

