使用multiprocessing调用函数时无print输出？如何修复？

串行调用示例 #

以下是串行调用函数的正常代码：

import time

start = time.perf_counter()

def do_something():
    print("Sleeping 1 second ...")
    time.sleep(1)
    print("Done Sleeping...\n")

do_something()
do_something()
do_something()

finish = time.perf_counter()

print("\n\n")
print(f"Finished in {round(finish-start,2)} second(s)")

预期输出：

Sleeping 1 second ...
    Done Sleeping...
    
    Sleeping 1 second ...
    Done Sleeping...
    
    Sleeping 1 second ...
    Done Sleeping...
    
    
    Finished in 3.07 second(s)

并行调用的问题 #

使用multiprocessing并行调用时，无法看到do_something中的print输出，代码如下：

import multiprocessing
import time

Nbre = multiprocessing.cpu_count()
print(f"Number of CPUs = {Nbre}")

start = time.perf_counter()

def do_something():
    print("Sleeping 1 second ...")
    time.sleep(1)
    print("Done Sleeping...\n")

p1 = multiprocessing.Process(target = do_something)
p2 = multiprocessing.Process(target = do_something)
p3 = multiprocessing.Process(target = do_something)

p1.start()
p2.start()
p3.start()

p1.join()
p2.join()
p3.join()

finish = time.perf_counter()

print("\n")
print(f"Finished in {round(finish-start,2)} second(s)")

实际输出：

Number of CPUs = 8
    
    
    Finished in 0.12 second(s)

原因分析 #

• 多进程环境中，子进程的标准输出默认采用全缓冲模式，而非终端环境的行缓冲。子进程结束时，输出缓存未及时刷新到主进程的输出流，导致内容丢失。
• Windows系统下使用multiprocessing时，未将主程序逻辑放在if __name__ == '__main__':块中，会导致子进程重复执行主代码，引发输出异常或进程创建错误。

修复方法 #

方法1：强制刷新输出缓存 #

在do_something的print语句后手动调用sys.stdout.flush()刷新缓存：

import multiprocessing
import time
import sys

def do_something():
    print("Sleeping 1 second ...")
    sys.stdout.flush()
    time.sleep(1)
    print("Done Sleeping...\n")
    sys.stdout.flush()

if __name__ == '__main__':
    Nbre = multiprocessing.cpu_count()
    print(f"Number of CPUs = {Nbre}")

    start = time.perf_counter()

    p1 = multiprocessing.Process(target=do_something)
    p2 = multiprocessing.Process(target=do_something)
    p3 = multiprocessing.Process(target=do_something)

    p1.start()
    p2.start()
    p3.start()

    p1.join()
    p2.join()
    p3.join()

    finish = time.perf_counter()

    print("\n")
    print(f"Finished in {round(finish-start,2)} second(s)")

方法2：使用print的flush参数（Python 3.3+） #

直接给print添加flush=True参数，无需导入sys模块：

import multiprocessing
import time

def do_something():
    print("Sleeping 1 second ...", flush=True)
    time.sleep(1)
    print("Done Sleeping...\n", flush=True)

if __name__ == '__main__':
    Nbre = multiprocessing.cpu_count()
    print(f"Number of CPUs = {Nbre}")

    start = time.perf_counter()

    p1 = multiprocessing.Process(target=do_something)
    p2 = multiprocessing.Process(target=do_something)
    p3 = multiprocessing.Process(target=do_something)

    p1.start()
    p2.start()
    p3.start()

    p1.join()
    p2.join()
    p3.join()

    finish = time.perf_counter()

    print("\n")
    print(f"Finished in {round(finish-start,2)} second(s)")

关键注意事项 #

• 必须用if __name__ == '__main__':包裹主程序逻辑，这是Windows系统下multiprocessing的强制要求，避免子进程重复初始化导致的异常。
• 刷新缓存后，子进程的print内容会实时输出到控制台，同时并行执行的总耗时会接近1秒（符合多进程预期）。

内容的提问来源于stack exchange，提问作者RACHID BEN ABDELMALEK