Docker部署Streamlit应用遇Scipy.sparse._csr模块找不到错误

错误信息 #

加载pickle模型时出现模块找不到错误：

File "main.py", line 14, in load_models
pickle.load(open("model.pk"), "rb")

ModuleNotFoundError: No module named 'scipy.sparse._csr'

相关配置文件 #

Dockerfile #

FROM python:3.6-slim

WORKDIR /app

RUN apt-get update
RUN apt-get install \
    'ffmpeg'\
    'libsm6'\
    'libxext6'  -y

RUN apt-get install rustc -y

RUN pip install numpy
RUN apt update && apt install -y build-essential && rm -rf /var/lib/apt/lists/*
RUN pip install scikit-image matplotlib more_itertools

COPY ./requirements.txt .

RUN pip install -r requirements.txt

COPY . .

RUN pip freeze > requiremets.txt

RUN cat requiremets.txt

EXPOSE 8000
EXPOSE 8501

CMD ["python", "main.py"]

requirements.txt（注：原标注为requirements.py，应为笔误） #

scipy<=1.5.4
classla
transformers
pickle-mixin
streamlit
scikit-learn

容器内已安装包（pip freeze输出） #

altair                4.1.0
argon2-cffi           21.3.0
argon2-cffi-bindings  21.2.0
async-generator       1.10
attrs                 22.2.0
backcall              0.2.0
backports.zoneinfo    0.2.1
bleach                4.1.0
blinker               1.5
cachetools            4.2.4
certifi               2022.12.7
cffi                  1.15.1
charset-normalizer    2.0.12
classla               1.1.0
click                 8.0.4
commonmark            0.9.1
cycler                0.11.0
dataclasses           0.8
decorator             4.4.2
defusedxml            0.7.1
entrypoints           0.4
filelock              3.4.1
gitdb                 4.0.9
GitPython             3.1.18
huggingface-hub       0.4.0
idna                  3.4
imageio               2.15.0
importlib-metadata    4.8.3
importlib-resources   5.4.0
ipykernel             5.5.6
ipython               7.16.3
ipython-genutils      0.2.0
ipywidgets            7.7.3
jedi                  0.17.2
Jinja2                3.0.3
joblib                1.1.1
jsonschema            3.2.0
jupyter-client        7.1.2
jupyter-core          4.9.2
jupyterlab-pygments   0.1.2
jupyterlab-widgets    1.1.2
kiwisolver            1.3.1
lxml                  4.9.2
MarkupSafe            2.0.1
matplotlib            3.3.4
mistune               0.8.4
more-itertools        8.14.0
nbclient              0.5.9
nbconvert             6.0.7
nbformat              5.1.3
nest-asyncio          1.5.6
networkx              2.5.1
notebook              6.4.10
numpy                 1.19.5
obeliks               1.1.6
packaging             21.3
pandas                1.1.5
pandocfilters         1.5.0
parso                 0.7.1
pexpect               4.8.0
pickle-mixin          1.0.2
pickleshare           0.7.5
Pillow                8.4.0
pip                   21.2.4
prometheus-client     0.16.0
prompt-toolkit        3.0.36
protobuf              3.19.6
ptyprocess            0.7.0
pyarrow               6.0.1
pycparser             2.21
pydeck                0.6.2
Pygments              2.14.0
Pympler               1.0.1
pyparsing             3.0.9
pyrsistent            0.18.0
python-dateutil       2.8.2
pytz                  2022.7.1
pytz-deprecation-shim 0.1.0.post0
PyWavelets            1.1.1
PyYAML                6.0
pyzmq                 25.0.0
regex                 2022.10.31
reldi-tokeniser       1.0.2
requests              2.27.1
rich                  12.6.0
sacremoses            0.0.53
scikit-image          0.17.2
scikit-learn          0.24.2
scipy                 1.5.4
semver                2.13.0
Send2Trash            1.8.0
setuptools            57.5.0
six                   1.16.0
smmap                 5.0.0
streamlit             1.10.0
terminado             0.12.1
testpath              0.6.0
threadpoolctl         3.1.0
tifffile              2020.9.3
tokenizers            0.12.1
toml                  0.10.2
toolz                 0.12.0
torch                 1.10.2
tornado               6.1
tqdm                  4.64.1
traitlets             4.3.3
transformers          4.18.0
typing_extensions     4.1.1
tzdata                2022.7
tzlocal               4.2
urllib3               1.26.14
validators            0.20.0
watchdog              2.3.0
wcwidth               0.2.6
webencodings          0.5.1
wheel                 0.37.0
widgetsnbextension    3.6.2
zipp                  3.6.0

解决方案 #

• 统一依赖管理，删除重复安装步骤：将numpy、scikit-image、matplotlib、more_itertools全部添加到requirements.txt中，删除Dockerfile里单独的pip install命令，避免依赖版本冲突或安装不完整。
• 补充Scipy编译依赖：在slim镜像中安装Scipy需要完整编译依赖才能确保所有模块安装成功，修改Dockerfile的系统依赖安装步骤：

  RUN apt-get update && apt-get install -y build-essential gfortran liblapack-dev libopenblas-dev ffmpeg libsm6 libxext6 rustc && rm -rf /var/lib/apt/lists/*
  

• 验证模型与Scipy版本兼容性：确认保存model.pk时使用的Scipy版本不超过1.5.4，不同版本的Scipy可能调整内部模块路径，导致加载pickle文件时找不到模块。
• 清理无效操作：删除Dockerfile中的RUN pip freeze > requiremets.txt和RUN cat requiremets.txt命令，减少镜像层并避免文件名笔误。
• 重新构建镜像：使用docker build --no-cache .重新构建，确保所有依赖从全新状态安装，避免缓存导致的问题。

内容的提问来源于stack exchange，提问作者Veneta