Docker+virtualenv环境下numpy加载npy文件报sklearn模块缺失错误

问题概述 #

在客户项目中，采用Docker结合Pipenv（基于virtualenv）管理运行环境，执行代码np.load(PATH_TO_NPY_FILE, allow_pickle=True)时，触发ModuleNotFoundError: No module named 'sklearn'错误。Pipfile中已声明依赖numpy = "==1.21.6"。

错误详情 #

错误触发代码行：

np.load(PATH_TO_NPY_FILE, allow_pickle=True)

对应的堆栈跟踪：

development_1  |   File "/root/.local/share/virtualenvs/code-_Py8Si6I/lib/python3.7/site-packages/numpy/lib/npyio.py", line 441, in load
development_1  |     pickle_kwargs=pickle_kwargs)
development_1  |   File "/root/.local/share/virtualenvs/code-_Py8Si6I/lib/python3.7/site-packages/numpy/lib/format.py", line 748, in read_array
development_1  |     array = pickle.load(fp, **pickle_kwargs)
development_1  | ModuleNotFoundError: No module named 'sklearn'

已尝试的解决方案 #

• 在Dockerfile中添加命令python -m pip install scikit-learn scipy matplotlib，手动安装scikit-learn
• 通过apt安装系统包python3-sklearn
  上述操作后，错误仍然存在。

项目相关信息 #

项目结构 #

|- src/
|  |- main.py
|
|- submodules/
|  |- module1
|  |  |- foo.py
|  |  |- setup.py
|  |
|  |- module2
|  |  |- bar.py
|  |
|  |- __init__.py
|
|- .gitmodules
|- Pipfile
|- Dockerfile

注：报错代码来自git子模块，顶层仓库有Pipfile，子模块无独立Pipfile，submodules文件夹包含__init__.py

Dockerfile内容 #

FROM python:3.7

WORKDIR code/

COPY Pipfile .
COPY submodules/ submodules/

RUN pip install pipenv && \
    pipenv install --deploy  && \
    python -m pip install scikit-learn scipy matplotlib && \
    apt-get update && \
    apt-get install -y locales ffmpeg libsm6 libxext6 libxrender-dev python3-sklearn && \
    sed -i -e 's/# nl_BE.UTF-8 UTF-8/nl_BE.UTF-8 UTF-8/' /etc/locale.gen && \
    dpkg-reconfigure --frontend=noninteractive locales

ENV LANG nl_BE.UTF-8
ENV LC_ALL nl_BE.UTF-8

COPY .env .
COPY src/ .
COPY data/ data

CMD [ "pipenv", "run", "python", "main.py" ]

Pipfile内容 #

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
python-dotenv = "*"
boto3 = "*"
pySqsListener = "*"
xpress = "==9.0.5"
module1 = {path = "./submodules/module1"}
pandas = "==1.3.4"
numpy = "==1.21.6"

[dev-packages]

[requires]
python_version = "3.7"

核心疑问 #

1. 为什么安装numpy不会自动安装sklearn？（误以为sklearn是numpy依赖树的一部分）
2. 明明手动安装了sklearn，为什么运行时还是出现模块找不到的错误？

内容的提问来源于stack exchange，提问作者gleerman