Docker容器中Python Multiprocessing的Manager运行失败问题

我用Docker容器运行流水线中的Python脚本，本地运行完全正常，但在容器中调用multiprocessing.Manager()时触发如下错误：

Process SyncManager-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.11/multiprocessing/managers.py", line 592, in _run_server
    server = cls._Server(registry, address, authkey, serializer)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/multiprocessing/managers.py", line 156, in __init__
    self.listener = Listener(address=address, backlog=16)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/multiprocessing/connection.py", line 447, in __init__
    self._listener = SocketListener(address, family, backlog)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/multiprocessing/connection.py", line 590, in __init__
    self._socket.bind(address)
OSError: [Errno 22] Invalid argument
Traceback (most recent call last):
  File "/source/ReferenceMatrix.py", line 113, in <module>
    with Manager() as manager:
         ^^^^^^^^^
  File "/usr/local/lib/python3.11/multiprocessing/context.py", line 57, in Manager
    m.start()
  File "/usr/local/lib/python3.11/multiprocessing/managers.py", line 567, in start
    self._address = reader.recv()
                    ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/multiprocessing/connection.py", line 249, in recv
    buf = self._recv_bytes()
          ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/multiprocessing/connection.py", line 413, in _recv_bytes
    buf = self._recv(4)
          ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/multiprocessing/connection.py", line 382, in _recv
    raise EOFError
EOFError

容器和本地环境均使用Python 3.11.0，Dockerfile基础镜像为python:3.11.0-slim。我已经在代码中用if __name__ == '__main__':包裹了Manager的调用逻辑：

if __name__ == '__main__':
  with Manager() as manager:
    ...

但错误仍然存在。

补充Dockerfile内容 #

# Build Preprocessing into clustered files 
## Base image
FROM python:3.11.0-slim

## Aquire date/time of build
ARG buildDate

## Retrive the library requirements text file and the python scripts for preprocessing of both references and samples
COPY requirements.txt /tmp/
ADD CreateRefMeta.py /source/
ADD SamplesMatrix.py /source/
ADD ReferenceMatrix.py /source/
ADD FormatSample.py /source/
ADD ClusterFilteringRef.py /source/
ADD support_functions.py /source/

################# METADATA #################
# Image information
LABEL about.image="Python base image with required libraries for MethAtlas"
LABEL image.authors="Maisa ------ Santos"
LABEL image.authors.email="-----@-----"
## Update when new python and libraries versions are available
LABEL image.maintainer="Maisa ----- Santos"
LABEL image.maintainer.email="-----@-----"
LABEL base_image="python:3.11.0-slim"
LABEL base_image.added_libraries="numpy==1.24.3, pandas==2.0.2, scipy==1.10.1, matplotlib==3.7.1, PyYAML==6.0"
LABEL base_image.other_software="bedtools==2.30"
LABEL image.version="1.0"
## Date/time of build
LABEL buildDate=${buildDate}
# Software information
LABEL software="MethAtlas for DNA methylation deconvolution"
LABEL software.version="No info available"
LABEL about.article="https://www.nature.com/articles/s41467-018-07466-6"
LABEL about.documentation="https://github.com/nloyfer/meth_atlas"
#LABEL about.license_file=""
#LABEL about.license=""
LABEL about.tags="Epigenomics,DNA methylation,Sequencing,Deconvolution"
# Docker command examples for running from this image
LABEL docker.run01.help="docker run --rm mferrosantos/preprocessing:1.0 /source/CreateRefMeta.py -h"
LABEL docker.run02.help="docker run --rm mferrosantos/preprocessing:1.0 /source/SamplesMatrix.py -h"
LABEL docker.run03.help="docker run --rm mferrosantos/preprocessing:1.0 /source/ReferenceMatrix.py -h"
LABEL docker.run04.help="docker run --rm mferrosantos/preprocessing:1.0 /source/FormatSample.py -h"
LABEL docker.run05.help="docker run --rm mferrosantos/preprocessing:1.0 /source/ClusterFilteringRef.py -h"
LABEL docker.run1.referenceList="docker run --rm -v path-to-data-directory:/mnt -it mferrosantos/preprocessing:1.0 /source/CreateRefMeta.py /mnt/config.yaml -o /mnt"
LABEL docker.run2.sampleMatrix="docker run --rm -v path-to-data-directory:/mnt -it mferrosantos/preprocessing:1.0 /source/SamplesMatrix.py /mnt/samples_path /mnt/path-to-clusterFile [-exp] [-cpg] -gen genomeVersion [-metaS] -o /mnt"
LABEL docker.run3.referenceMatrix="docker run --rm -v path-to-data-directory:/mnt -it mferrosantos/preprocessing:1.0 /source/ReferenceMatrix.py /mnt/path-to-RefMeta -gen genomeVersion -re /mnt/path-to-the-resource-folder -o /mnt"
LABEL docker.run4.formatSamples="docker run --rm -v path-to-data-directory:/mnt -it mferrosantos/preprocessing:1.0 /source/FormatSample.py /mnt/path-to-sampleMatrix -o /mnt"
LABEL docker.run5.clusterFilterRefs="docker run --rm -v path-to-data-directory:/mnt -it mferrosantos/preprocessing:1.0 /source/ClusterFilteringRef.py /mnt/path-to-refMatrix [-DMR DMRmethod] [-cluster DMRclusterFile] -it iterations -col refCollapse [-ex [exclude1 exclude2 ...]] -cF clusterFiltering [-r randomSeed] -o /mnt"

############################################

## Check for updates to the system and install them
RUN apt-get update -y && apt-get install -y\
 libcurl4-gnutls-dev \
 libssl-dev \
 libssh2-1-dev \
 libxml2-dev \
 libhdf5-dev \
 libgmp-dev \
 libpng-dev \
 libgsl-dev \
 libxt-dev \
 libcairo2-dev \
 libtiff-dev \
 fftw-dev \
 procps \
 python3-pip

## Update pip
RUN /usr/local/bin/python -m pip install --upgrade pip

## Install python requirements
RUN pip install --no-cache-dir -r /tmp/requirements.txt

## Install bedtools
RUN apt-get install bedtools

## Image uses the python3 command line as the entry point
ENTRYPOINT [ "python3" ]

出问题的脚本是ReferenceMatrix.py，该脚本在Mac、Linux和Windows系统上均可正常运行，其他不使用Manager()的脚本在容器中也能正常输出结果，因此怀疑问题出在Docker镜像配置上。

解决方法 #

经过多次测试，将基础镜像从python:3.11.0-slim更换为python:3.11.4后，问题完全解决。唯一的缺点是镜像体积有所增大。

内容的提问来源于stack exchange，提问作者Maisa Santos