Docker镜像构建异常：Ubuntu18.04本地无法安装numpy但Sagemaker实例正常

基于AWS的Scikit自定义容器示例，使用下方Dockerfile构建镜像时，在Sagemaker笔记本实例上可正常运行，但本地构建时出现numpy相关错误。尝试更换numpy版本、安装Python3.8、改用Ubuntu20.04等方法，均引发其他错误。疑惑点：

• 为何完全相同的Dockerfile在不同环境下会产生不同错误？容器本该从同一源下载内容并一致构建，这是容器的核心优势。
• 已确认两种环境安装的Python版本相同，且在笔记本实例运行镜像时未出现需Python3.8的警告，为何numpy安装行为存在差异？

错误日志 #

Collecting numpy==1.16.2
  Downloading https://files.pythonhosted.org/packages/cf/8d/6345b4f32b37945fedc1e027e83970005fc9c699068d2f566b82826515f2/numpy-1.16.2.zip (5.1MB)
Collecting scipy==1.2.1
  Downloading https://files.pythonhosted.org/packages/a9/b4/5598a706697d1e2929eaf7fe68898ef4bea76e4950b9efbe1ef396b8813a/scipy-1.2.1.tar.gz (23.1MB)
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "/usr/lib/python3/dist-packages/setuptools/sandbox.py", line 154, in save_modules
        yield saved
      File "/usr/lib/python3/dist-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/usr/lib/python3/dist-packages/setuptools/sandbox.py", line 250, in run_setup
        _execfile(setup_script, ns)
      File "/usr/lib/python3/dist-packages/setuptools/sandbox.py", line 45, in _execfile
        exec(code, globals, locals)
      File "/tmp/easy_install-ua250hpf/numpy-1.24.2/setup.py", line 20, in <module>
    
    RuntimeError: Python version >= 3.8 required.

Dockerfile内容 #

# Build an image that can do training and inference in SageMaker
# This is a Python 3 image that uses the nginx, gunicorn, flask stack
# for serving inferences in a stable way.

FROM ubuntu:18.04

MAINTAINER Amazon AI <sage-learner@amazon.com>


RUN apt-get -y update && apt-get install -y --no-install-recommends \
         wget \
         python3-pip \
         python3-setuptools \
         nginx \
         ca-certificates \
    && rm -rf /var/lib/apt/lists/*

RUN python3 --version
RUN ln -s /usr/bin/python3 /usr/bin/python
RUN ln -s /usr/bin/pip3 /usr/bin/pip

# Here we get all python packages.
# There's substantial overlap between scipy and numpy that we eliminate by
# linking them together. Likewise, pip leaves the install caches populated which uses
# a significant amount of space. These optimizations save a fair amount of space in the
# image, which reduces start up time.
RUN pip --no-cache-dir install numpy==1.16.2 scipy==1.2.1 scikit-learn==0.20.2 pandas flask gunicorn

# Set some environment variables. PYTHONUNBUFFERED keeps Python from buffering our standard
# output stream, which means that logs can be delivered to the user quickly. PYTHONDONTWRITEBYTECODE
# keeps Python from writing the .pyc files which are unnecessary in this case. We also update
# PATH so that the train and serve programs are found when the container is invoked.

ENV PYTHONUNBUFFERED=TRUE
ENV PYTHONDONTWRITEBYTECODE=TRUE
ENV PATH="/opt/program:${PATH}"

# Set up the program in the image
COPY decision_trees /opt/program
WORKDIR /opt/program

核心原因 #

错误日志显示：安装scipy==1.2.1时，内部触发easy_install拉取了numpy==1.24.2，而该版本numpy要求Python≥3.8，但Ubuntu18.04默认Python3版本为3.6，导致冲突。两种环境结果不同的本质是pip/setuptools版本差异：

• Sagemaker笔记本实例的内部源提供的python3-pip/python3-setuptools版本较低，依赖解析逻辑不会触发这种错误；
• 本地环境的setuptools版本过高（通常是58.x及以上），对旧版Python包的依赖解析逻辑发生变化，错误地为scipy 1.2.1解析出不兼容的高版本numpy。

解决办法 #

1. 锁定pip/setuptools版本：在安装其他包前，先安装兼容旧版包的工具链版本，修改Dockerfile中的包安装步骤：

   # 先安装兼容的pip和setuptools版本
   RUN pip install --upgrade pip==20.3.4 setuptools==57.5.0
   # 再安装业务依赖包
   RUN pip --no-cache-dir install numpy==1.16.2 scipy==1.2.1 scikit-learn==0.20.2 pandas flask gunicorn
   

2. 隔离scipy构建环境：先单独安装numpy，再给scipy添加--no-build-isolation参数，避免构建时拉取新的numpy：

   RUN pip --no-cache-dir install numpy==1.16.2
   RUN pip --no-cache-dir install --no-build-isolation scipy==1.2.1 scikit-learn==0.20.2 pandas flask gunicorn
   

3. 验证基础包版本：在Dockerfile中添加查看setuptools版本的命令，确保环境一致：

   RUN pip show setuptools
   

内容的提问来源于stack exchange，提问作者L Xandor