构建含detectron2的Python3.10版AWS Lambda遇torch模块缺失问题求助

问题描述 #

我正在构建集成Detectron2的Python 3.10版AWS Lambda，但始终无法成功。Dockerfile内容如下：

FROM public.ecr.aws/lambda/python:3.10 AS build

ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
ENV LAMBDA_TASK_ROOT /var/task

WORKDIR ${LAMBDA_TASK_ROOT} 

RUN yum update -y \
&& yum install -y \
git \
gcc \
gcc-c++ \
make \
cmake \
wget \
libpng-devel \
libjpeg-devel \
libtiff-devel \
zlib-devel \
mesa-libGL \
mesa-libGL-devel \
opencv \
tar \
&& yum clean all

RUN pip install --upgrade pip 

RUN pip install torch torchvision 

RUN pip install 'git+https://github.com/facebookresearch/detectron2.git' 

RUN pip install cython pycocotools 

RUN pip install --target "${LAMBDA_TASK_ROOT}" torch torchvision detectron2 cython pycocotools 

COPY requirements.txt . 
RUN pip install --target "${LAMBDA_TASK_ROOT}" -r requirements.txt

RUN rm -rf /var/cache/yum && rm -rf /tmp/* 

EXPOSE 8080 

CMD ["app.lambda_handler"] 

执行安装Detectron2的步骤时，出现了"No module named torch"的错误。

错误原因分析 #

问题出在依赖安装的路径和顺序上：AWS Lambda的Python运行环境仅从/var/task目录加载模块，而你先全局安装了torch，之后又将torch重新安装到/var/task，但安装Detectron2的步骤是在全局环境执行，此时全局环境的torch路径未被正确识别，导致Detectron2安装时找不到依赖。

解决方案 #

修正后的Dockerfile #

调整安装逻辑，所有依赖直接安装到${LAMBDA_TASK_ROOT}，并指定适配Lambda架构的torch版本，避免源码编译问题：

FROM public.ecr.aws/lambda/python:3.10 AS build

ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
ENV LAMBDA_TASK_ROOT /var/task
# 确保target目录下的模块和命令能被pip识别
ENV PATH="${LAMBDA_TASK_ROOT}/bin:${PATH}"
ENV PYTHONPATH="${LAMBDA_TASK_ROOT}:${PYTHONPATH}"

WORKDIR ${LAMBDA_TASK_ROOT} 

# 安装系统依赖并清理缓存
RUN yum update -y \
&& yum install -y \
git \
gcc \
gcc-c++ \
make \
cmake \
wget \
libpng-devel \
libjpeg-devel \
libtiff-devel \
zlib-devel \
mesa-libGL \
mesa-libGL-devel \
opencv \
tar \
&& yum clean all \
&& rm -rf /var/cache/yum /tmp/*

RUN pip install --upgrade pip 

# 安装适配x86_64 Linux的torch和torchvision，直接指定target目录
RUN pip install --target "${LAMBDA_TASK_ROOT}" torch==2.0.1 torchvision==0.15.2 --index-url https://download.pytorch.org/whl/cpu

# 此时pip能找到target目录下的torch，正常安装detectron2
RUN pip install --target "${LAMBDA_TASK_ROOT}" 'git+https://github.com/facebookresearch/detectron2.git'

# 安装其他依赖
RUN pip install --target "${LAMBDA_TASK_ROOT}" cython pycocotools

COPY requirements.txt . 
RUN pip install --target "${LAMBDA_TASK_ROOT}" -r requirements.txt

EXPOSE 8080 

CMD ["app.lambda_handler"] 

额外注意事项 #

• 检查requirements.txt，避免重复依赖导致版本冲突
• 如果Detectron2安装仍失败，可指定具体版本，比如git+https://github.com/facebookresearch/detectron2.git@v0.6
• Lambda解压后存储空间上限为250MB，可清理torch缓存、编译残留文件来缩小镜像体积
• 测试时可进入镜像shell，执行python -c "import torch; import detectron2"验证依赖是否正确安装

内容的提问来源于stack exchange，提问作者Narayanan_Symbiance