基于Airflow的Python项目Docker部署报错：celery不在$PATH中

环境与问题详情 #

• 自定义Dockerfile：

# BASE IMAGE
FROM 524881529748.dkr.ecr.ap-south-1.amazonaws.com/hotels-ecs-base:python3.6

ENV APAXY http://apaxy.mmt.com/

RUN yum-config-manager --add-repo http://bizeye.mmt.com/SOFT/nexus.repo && yum-config-manager --disable \* && yum-config-manager --enable Nexus3

RUN yum remove -y mysql-community-server-minimal
RUN yum install -y gcc
RUN yum install -y mysql-devel

# INSTALL ANACONDA
RUN yum install -y wget
RUN wget https://repo.continuum.io/archive/Anaconda2-2019.07-Linux-x86_64.sh -O ~/anaconda.sh
RUN /bin/bash ~/anaconda.sh -b -p /opt/conda
RUN yes | rm ~/anaconda.sh

# ACTIVATE CONDA & INSTALL REQUIREMENTS
COPY docker-requirements.txt docker-requirements.sh /tmp/
RUN sh /tmp/docker-requirements.sh

USER airflow

• 镜像构建成功，命令：docker build . -f Dockerfile --pull --tag hotels-ecs-base:python3.6
• 使用Airflow官方2.3.3版本docker-compose.yaml启动，报错：

[+] Running 6/6
 ⠿ Container airflow-compose-postgres-1           Running                                                                                                                                                  0.0s
 ⠿ Container airflow-compose-redis-1              Running                                                                                                                                                  0.0s
 ⠿ Container airflow-compose-airflow-worker-1     Recreated                                                                                                                                                0.1s
 ⠿ Container airflow-compose-airflow-webserver-1  Recreated                                                                                                                                                0.1s
 ⠿ Container airflow-compose-airflow-scheduler-1  Recreated                                                                                                                                                0.1s
 ⠿ Container airflow-compose-flower-1             Recreated                                                                                                                                                0.1s
Attaching to airflow-compose-airflow-init-1, airflow-compose-airflow-scheduler-1, airflow-compose-airflow-webserver-1, airflow-compose-airflow-worker-1, airflow-compose-flower-1, airflow-compose-postgres-1, airflow-compose-redis-1
Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "celery": executable file not found in $PATH: unknown

错误原因 #

1. Python版本不兼容：安装的Anaconda2对应Python2.7，但Airflow 2.x要求Python3.6+，导致依赖安装异常
2. 环境变量未配置：conda的bin目录未加入系统$PATH，即使安装了celery，容器启动时也找不到可执行文件
3. 用户切换后的环境问题：切换到airflow用户后，未加载conda的环境变量，导致命令无法识别

解决方案 #

1. 替换Anaconda为Python3版本 #

Airflow 2.3.3支持Python3.6~3.10，修改Dockerfile中的Anaconda安装步骤：

# 替换原Anaconda安装代码
RUN yum install -y wget
RUN wget https://repo.continuum.io/archive/Anaconda3-2023.07-2-Linux-x86_64.sh -O ~/anaconda.sh
RUN /bin/bash ~/anaconda.sh -b -p /opt/conda
RUN yes | rm ~/anaconda.sh

2. 配置全局环境变量 #

在Dockerfile中添加conda路径到$PATH，确保所有用户都能访问：

# 安装Anaconda后添加
ENV PATH="/opt/conda/bin:$PATH"

3. 确认依赖完整性 #

确保docker-requirements.txt包含Airflow和Celery相关依赖：

apache-airflow==2.3.3
apache-airflow[celery]==2.3.3
redis==4.3.4
celery==5.2.7

4. 验证依赖安装（可选调试步骤） #

在Dockerfile末尾添加验证命令，确保celery安装成功：

RUN celery --version

5. 重新构建并启动 #

执行以下命令重新构建镜像并启动容器：

docker build . -f Dockerfile --pull --tag hotels-ecs-base:python3.6
docker-compose up

内容的提问来源于stack exchange，提问作者Mohit