Docker中FastAPI的Python多进程子进程无法启动问题问询

我在配备8核CPU的Docker容器中运行FastAPI，希望通过Python multiprocessing 模块利用多核资源。容器外直接用uvicorn运行程序时一切符合预期，但构建并启动容器后，子进程无法正常启动。

相关代码 #

import multiprocessing
from multiprocessing import Process, Manager, Queue
from contextlib import asynccontextmanager
from fastapi import FastAPI

CLIENT_CORES = 5
connection = "mongo_connection_string"

@asynccontextmanager
async def lifespan(app: FastAPI):
    
    manager = multiprocessing.Manager()

    global dbWorkers
    global clientWorkers

    global clientQueue
    global clientResponse

    # resources
    dbQueue = Queue()
    clientQueue = Queue()
    clientResponse = manager.dict()
    psetObjects = manager.dict()
    dbWorkers = []
    clientWorkers = []

    # start client and db workers
    config_processes(dbQueue,clientQueue,clientResponse,psetObjects)

    yield
    
    graceful_shutdown()

def config_processes(dbQueue,clientQueue,clientResponse,psetObjects):
    # db
    start_db_workers(dbQueue,psetObjects)
    # client
    start_client_workers(clientQueue,psetObjects,dbQueue,clientResponse)

def start_db_workers(dbQueue,psetObjects):
    print("Building db Workers")

    for p in range(DB_PROCS):
        dbworker = Process(target=db_worker,args=(connection,dbQueue,psetObjects,p))
        dbWorkers.append(dbworker)
        dbworker.start()

    print("db Pool of {} db Worker(s) Built -OK".format(DB_PROCS))

def start_client_workers(clientQueue,psetObjects,dbQueue,clientResponse):
    print("Building client Workers")
    print("CLIENT CORES: {}".format(CLIENT_CORES))
    
    for p in range(CLIENT_CORES):
        clientWorker = Process(target=client_worker,args=(clientQueue,psetObjects,dbQueue,clientResponse,p),name="client-worker-{}".format(p))
        clientWorkers.append(clientWorker)
        clientWorker.start()
    
    print("Client Pool of {} Worker(s) Built -OK".format(len(clientWorkers)))

def db_worker(connection,dbQueue,pset_objects,n):
    dbworker = DbWorker(connection,dbQueue,pset_objects,n)
    print("built dbworker {}".format(n))
    dbworker.listen()

def client_worker(clientQueue,psetObjects,dbQueue,clientResponse,p):
    clientworker = ClientWorker(clientQueue,psetObjects,dbQueue,clientResponse,p)
    print("built client worker {}".format(p))
    clientworker.listen()

容器内外运行日志对比 #

Docker容器内运行日志 #

INFO:     Started server process [7]
INFO:     Waiting for application startup.
CPUS: 8
Building db Workers
db Pool of 1 db Worker(s) Built -OK
Building client Workers
CLIENT CORES: 5
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:80 (Press CTRL+C to quit)

注：从未看到client_worker()和db_worker()目标函数的打印输出，且start_client_workers()未执行完成（没有输出"Client Pool of 5 Worker(s) Built -OK"）

容器外直接用uvicorn运行日志 #

CPUS: 8
INFO:     Started server process [40230]
INFO:     Waiting for application startup.
Building db Workers
db Pool of 1 db Worker(s) Built -OK
Building client Workers
CLIENT CORES: 5
Client Pool of 5 Worker(s) Built -OK
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
built client worker 4
built client worker 2
built client worker 1
built client worker 0
built client worker 3
built dbworker 0

后续测试补充（2024年2月12日） #

在全新环境中按方案实现后，出现如下异常：

INFO:     Started server process [1]
INFO:     Waiting for application startup.
Building db Workers
db Pool of 2 db Worker(s) Built -OK
Building client Workers
CLIENT CORES: 5
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
built client worker 0
built dbworker 1
built client worker 1
built dbworker 0
built client worker 3
built client worker 4
built client worker 2

注：能看到client_worker()的所有打印语句，但看不到print(f"Client Pool of {len(clientWorkers)} Worker(s) Built -OK")的输出，和原环境现象一致——最后一次Process.start()调用似乎未返回，且无论CLIENT_CORES设为何值，最后一次调用都不会返回。

添加完整业务逻辑后的日志 #

当启用ClientWorker.listen()方法时，日志如下：

INFO:     Started server process [7]
INFO:     Waiting for application startup.
CPUS: 8
Building db Workers
db Pool of 1 db Worker(s) Built -OK
Building client Workers
CLIENT CORES: 5
built dbworker 0
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:80 (Press CTRL+C to quit)

注释listen()调用后的日志 #

如果注释掉clientworker.listen()：

def client_worker(clientQueue,psetObjects,dbQueue,clientResponse,p):
    clientworker = ClientWorker(clientQueue,psetObjects,dbQueue,clientResponse,p)
    print("built client worker {}".format(p))
    # clientworker.listen()

日志变为：

INFO:     Started server process [6]
INFO:     Waiting for application startup.
CPUS: 8
Building db Workers
db Pool of 1 db Worker(s) Built -OK
Building client Workers
CLIENT CORES: 5
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:80 (Press CTRL+C to quit)
built client worker 0
built dbworker 0
built client worker 2
built client worker 1
built client worker 3
built client worker 4

ClientWorker.listen()方法代码 #

def listen(self):
    print("started")
    while self.stay_alive == True:
        try:
            args = self.clientQueue.get(block=True)
            # do_work(args)
        except:
           # report_error()

疑问 #

仅用uvicorn运行时一切正常，但Docker环境中出现上述异常。请问：

1. 能否通过现有方法实现容器内的预期行为？
2. 如果需要重构，是否要改用Gunicorn、Docker Swarm等方案？若需重构，请给出正确实现方式。

内容的提问来源于stack exchange，提问作者Nahko