远程服务器容器运行Jupyter Notebook遇urllib3模块缺失错误

在远程服务器通过Apptainer容器运行Jupyter Notebook时，触发ModuleNotFoundError: No module named 'urllib3.util.wait'错误。已确认容器和服务器计算节点中均存在urllib3包，执行以下命令后报错：

[u102501@mel0158 get_BIO1018]$ module load Apptainer/1.2.4-GCCcore-12.3.0
[u102501@mel0158 get_BIO1018]$ apptainer exec --bind /project/home/xxxxxx/get_BIO1018:/home/xf2217 get_complete_latest.sif jupyter notebook --allow-root --ip 0.0.0.0 --no-browser   --NotebookApp.token='' --NotebookApp.password=''   --notebook-dir=/home/xf2217

报错堆栈信息：

Traceback (most recent call last):
  File "/opt/conda/bin/jupyter-lab", line 6, in <module>
    from jupyterlab.labapp import main
  File "/opt/conda/lib/python3.12/site-packages/jupyterlab/__init__.py", line 8, in <module>
    from .handlers.announcements import (
  File "/opt/conda/lib/python3.12/site-packages/jupyterlab/handlers/announcements.py", line 15, in <module>
    from jupyterlab_server.translation_utils import translator
  File "/opt/conda/lib/python3.12/site-packages/jupyterlab_server/__init__.py", line 6, in <module>
    from .app import LabServerApp
  File "/opt/conda/lib/python3.12/site-packages/jupyterlab_server/app.py", line 15, in <module>
    from .handlers import LabConfig, add_handlers
  File "/opt/conda/lib/python3.12/site-packages/jupyterlab_server/handlers.py", line 21, in <module>
    from .listings_handler import ListingsHandler, fetch_listings
  File "/opt/conda/lib/python3.12/site-packages/jupyterlab_server/listings_handler.py", line 10, in <module>
    import requests
  File "/opt/conda/lib/python3.12/site-packages/requests/__init__.py", line 43, in <module>
    import urllib3
  File "/home/users/u102501/.local/lib/python3.12/site-packages/urllib3/__init__.py", line 13, in <module>
    from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool, connection_from_url
  File "/home/users/u102501/.local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 13, in <module>
    from .connection import (
  File "/home/users/u102501/.local/lib/python3.12/site-packages/urllib3/connection.py", line 15, in <module>
    from .util.proxy import create_proxy_ssl_context
  File "/home/users/u102501/.local/lib/python3.12/site-packages/urllib3/util/__init__.py", line 4, in <module>
    from .connection import is_connection_dropped
  File "/home/users/u102501/.local/lib/python3.12/site-packages/urllib3/util/connection.py", line 8, in <module>
    from .wait import NoWayToWaitForSocketError, wait_for_read
ModuleNotFoundError: No module named 'urllib3.util.wait'
srun: error: mel0158: task 0: Exited with exit code 1

已尝试在容器内外重装urllib3，但问题未解决。服务器仅支持Singularity/Apptainer，无法使用Docker。

解决方案 #

1. 隔离宿主环境变量 #

从报错路径能看到，Python优先加载了服务器用户本地目录的urllib3，而非容器内版本，导致版本不兼容。运行Apptainer时添加--cleanenv参数清除宿主环境变量，避免加载外部依赖：

apptainer exec --cleanenv --bind /project/home/xxxxxx/get_BIO1018:/home/xf2217 get_complete_latest.sif jupyter notebook --allow-root --ip 0.0.0.0 --no-browser --NotebookApp.token='' --NotebookApp.password='' --notebook-dir=/home/xf2217

2. 直接调用容器内Python解释器 #

如果--cleanenv影响其他必要环境，可直接指定容器内的Python路径，确保使用容器内的依赖库：

apptainer exec --bind /project/home/xxxxxx/get_BIO1018:/home/xf2217 get_complete_latest.sif /opt/conda/bin/python -m jupyter notebook --allow-root --ip 0.0.0.0 --no-browser --NotebookApp.token='' --NotebookApp.password='' --notebook-dir=/home/xf2217

3. 重装容器内适配版本的urllib3 #

进入容器内部，重新安装与容器内Python、requests版本匹配的urllib3：

apptainer exec --bind /project/home/xxxxxx/get_BIO1018:/home/xf2217 get_complete_latest.sif /opt/conda/bin/pip install --force-reinstall urllib3==2.1.0

注：版本号可根据容器内Python 3.12的兼容性调整，推荐使用2.x系列稳定版

4. 临时禁用用户本地包加载 #

运行命令前设置环境变量，让Python忽略用户本地的包目录：

export PYTHONNOUSERSITE=1
apptainer exec --bind /project/home/xxxxxx/get_BIO1018:/home/xf2217 get_complete_latest.sif jupyter notebook --allow-root --ip 0.0.0.0 --no-browser --NotebookApp.token='' --NotebookApp.password='' --notebook-dir=/home/xf2217

内容的提问来源于stack exchange，提问作者Faith Ogundimu