Python3环境下Jupyter Notebook调用multiprocessing却用Python2.7的问题

在Python3环境的Jupyter Notebook中，尝试使用concurrent.futures.ProcessPoolExecutor并行处理字典字符串两两比对逻辑（masterprova为包含100个元素的字典），前期分块代码运行正常，但调用多进程合并分块时，触发了Python2.7版本multiprocessing模块的语法错误。

相关代码 #

分块处理代码 #

import concurrent.futures
from collections import defaultdict
from itertools import combinations
#import fuzz

def process_chunk(masterprova_chunk, d, threshold):
    for (i,ii),(j,jj) in combinations(masterprova_chunk, 2):
        if jj not in d[i] and fuzz.partial_ratio(ii,jj) >= threshold:
            d[i].add(jj)
    return d

threshold = 90
d = defaultdict(set)

for (i,ii) in master.items():
    d[i].add(ii)
    
# Split masterprova into chunks for parallel processing
num_workers = 4 # number of parallel processes to use
chunk_size = len(masterprova)//num_workers
masterprova_chunks = [list(masterprova.items())[i:i + chunk_size] for i in range(0, len(masterprova), chunk_size)]

多进程调用代码 #

with concurrent.futures.ProcessPoolExecutor(max_workers=num_workers) as executor:
    results = [executor.submit(process_chunk, masterprova_chunk, d, threshold) for masterprova_chunk in masterprova_chunks]
    for f in concurrent.futures.as_completed(results):
        d.update(f.result())

错误信息 #

Traceback (most recent call last):

  File ~/opt/anaconda3/envs/single_auth_paper/lib/python3.10/site-packages/IPython/core/interactiveshell.py:3397 in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  Input In [53] in <cell line: 1>
    with concurrent.futures.ProcessPoolExecutor(max_workers=num_workers) as executor:

  File ~/opt/anaconda3/envs/single_auth_paper/lib/python3.10/concurrent/futures/__init__.py:44 in __getattr__
    from .process import ProcessPoolExecutor as pe

  File ~/opt/anaconda3/envs/single_auth_paper/lib/python3.10/concurrent/futures/process.py:51 in <module>
    import multiprocessing as mp

  File /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/__init__.py:64 in <module>
    from multiprocessing.process import Process, current_process, active_children

  File /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py:271
    except SystemExit, e:
           ^
SyntaxError: multiple exception types must be parenthesized

1. 清理Python模块搜索路径，移除Python2.7相关路径 #

• 先在Notebook单元格中运行以下代码，检查当前环境的模块搜索路径：

  import sys
  print(sys.path)
  

• 若输出中包含/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/这类Python2.7的路径，说明路径被污染，执行以下代码移除：

  import sys
  sys.path = [path for path in sys.path if "Python2.7" not in path]
  

• 若使用conda环境，确保已激活目标Python3环境，且系统环境变量未混入Python2的路径。

2. 修正多进程数据传递逻辑，避免直接传递不可序列化对象 #

原代码直接将defaultdict对象d传入子进程，会导致对象复制、同步问题，甚至序列化失败。修改为子进程返回独立结果片段，主进程统一合并：

修改后的process_chunk函数 #

def process_chunk(masterprova_chunk, threshold):
    chunk_result = defaultdict(set)
    # 初始化当前分块内的键值
    for i, ii in masterprova_chunk:
        chunk_result[i].add(ii)
    # 处理分块内的两两比对
    for (i, ii), (j, jj) in combinations(masterprova_chunk, 2):
        if fuzz.partial_ratio(ii, jj) >= threshold:
            chunk_result[i].add(jj)
            chunk_result[j].add(ii)
    return chunk_result

修改后的多进程调用代码 #

threshold = 90
d = defaultdict(set)

# 初始化主字典的基础值
for i, ii in masterprova.items():
    d[i].add(ii)

num_workers = 4
chunk_size = len(masterprova) // num_workers
masterprova_chunks = [list(masterprova.items())[i:i+chunk_size] for i in range(0, len(masterprova), chunk_size)]

with concurrent.futures.ProcessPoolExecutor(max_workers=num_workers) as executor:
    # 仅传递分块数据和阈值，不传递主字典
    results = [executor.submit(process_chunk, chunk, threshold) for chunk in masterprova_chunks]
    for f in concurrent.futures.as_completed(results):
        chunk_res = f.result()
        # 合并子进程结果到主字典
        for key, values in chunk_res.items():
            d[key].update(values)

3. 确认Jupyter Notebook使用正确的Python3内核 #

在Jupyter界面顶部菜单中，选择Kernel -> Change Kernel，切换到对应Python3版本的conda环境内核，避免误选Python2内核。

4. 清理错误安装的第三方multiprocessing包 #

multiprocessing是Python标准库，无需通过pip安装/卸载。若之前错误安装了第三方版本，执行以下操作清理：

# 检查是否存在第三方multiprocessing包
pip3 list | grep multiprocessing
# 若存在，卸载第三方包
pip3 uninstall -y multiprocessing

内容的提问来源于stack exchange，提问作者Lusian