Python多进程随CPU数量增加运行变慢的问题咨询

数据老陈

2026-4-14

我正在尝试并行化一段本应是易并行化的代码，但奇怪的是，使用的进程越多，运行速度反而越慢。下面是一个最小复现的（有问题的）示例代码：

import os
import time
import random
import multiprocessing
from multiprocessing import Pool, Manager, Process

import numpy as np
import pandas as pd


def pool_func(
    number: int,
    max_number: int
) -> dict:
    
    pid = str(multiprocessing.current_process().pid)
    
    print('[{:2d}/{:2d} {:s}] Starting ...'.format(number, max_number, pid))
    t0 = time.time()
    # # the following takes ~10 seconds on a separate node
    # for i in range(2):
    #     print('[{:d}] Passed loop {:d}/2...'.format(number, i+1))
    #     time.sleep(5)
    
    # the following takes ~3.3 seconds on a separate node
    n = 1000
    for _ in range(50):
        u = np.random.randn(n, n)
        v = np.linalg.inv(u)
    
    t1 = time.time()
    print('[{:2d}/{:2d} {:s}] Finished in {:.1f} seconds.'.format(number, max_number, pid, t1 - t0))
    return {}
    

if __name__ == "__main__":
    runs = []
    count = 0
    while count < 50:
        runs.append(
            (count, 50)
        )
        count += 1
    
    print(f"Number of runs to perform: {len(runs):d}")
    
    num_cpus = 4
    print(f"Running job with {num_cpus:d} CPUs in parallel ...")
    
    # with Pool(processes=num_cpus) as pool:
    with multiprocessing.get_context("spawn").Pool(processes=num_cpus) as pool:
        results = pool.starmap(pool_func, runs)
    
    print('Main process done.')

我想指出三个关键点：

可以修改num_cpus的值来增加进程池中的工作进程数量
可以从默认的fork进程池切换到spawn方法，但这似乎没有任何变化
在pool_func内部，运行的任务可以是CPU密集型的矩阵求逆，也可以是不占用CPU的等待函数

当我使用等待函数时，进程的运行时间符合预期，每个进程大约耗时10秒。但当我使用矩阵求逆任务时，单个进程的运行时间会随着进程数量的增加而变长，大致数据如下：

1 CPU :  3 seconds  
2 CPUs:  4 seconds  
4 CPUs: 30 seconds  
8 CPUs: 95 seconds

以下是上述脚本运行时的部分输出：

Number of runs to perform: 50
Running job with 4 CPUs in parallel ...
[ 0/50 581194] Starting ...
[ 4/50 581193] Starting ...
[ 8/50 581192] Starting ...
[12/50 581191] Starting ...
[ 0/50 581194] Finished in 24.7 seconds.
[ 1/50 581194] Starting ...
[ 4/50 581193] Finished in 29.3 seconds.
[ 5/50 581193] Starting ...
[12/50 581191] Finished in 30.3 seconds.
[13/50 581191] Starting ...
[ 8/50 581192] Finished in 32.2 seconds.
[ 9/50 581192] Starting ...
[ 1/50 581194] Finished in 26.9 seconds.
[ 2/50 581194] Starting ...
[ 5/50 581193] Finished in 30.3 seconds.
[ 6/50 581193] Starting ...
[13/50 581191] Finished in 30.8 seconds.
[14/50 581191] Starting ...
[ 9/50 581192] Finished in 32.8 seconds.
[10/50 581192] Starting ...
...

我看到进程ID都是唯一的。

显然，这里存在扩展性问题——增加CPU数量反而导致单个进程运行变慢。被计时的进程中完全没有I/O操作，这些都是很常规的任务，我原本以为可以直接正常运行。我完全不知道为什么实际情况和预期不符，为什么这个脚本中单个进程的运行时间会随着CPU数量的增加而变长？

另外补充：在我的macOS笔记本上运行时，结果符合预期，但在我访问的另一台远程Linux服务器上出现了类似的扩展性问题。这可能是平台相关的问题，但我还是把问题提出来，希望有人遇到过类似情况并知道解决方法。

备注：内容来源于stack exchange，提问作者Finncent Price