为何大型Base64字符串单线程解码比多线程/多进程更快？

我有多段大型Base64字符串需要解码，大小从几百MB到约5GB不等。常规方案是直接调用base64.b64decode（基准实现），但尝试用多进程/多线程加速解码时，发现速度远慢于基准实现。

测试结果如下：

reference_implementation
decoding time = 7.37

implmementation1
Verify result Ok
decoding time = 7.59

threaded_impl
Verify result Ok
decoding time = 13.24

mutiproc_impl
Verify result Ok
decoding time = 11.82

请问我哪里操作有误？

（警告：该代码占用大量内存！）

import base64

from time import perf_counter
from binascii import a2b_base64
import concurrent.futures as fut
from time import sleep
from gc import collect
from multiprocessing import cpu_count

def reference_implementation(encoded):
    """This is the implementation that gives the desired result"""
    return base64.b64decode(encoded)


def implmementation1(encoded):
    """Try to call the directly the underlying library"""
    return a2b_base64(encoded)


def threaded_impl(encoded, N):
    """Try multi threading calling the underlying library"""
    # split the string into pieces
    d = len(encoded) // N            # number of splits
    lbatch = (d // 4) * 4           # lenght of first N-1 batches, the last is len(source) - lbatch*N
    batches = []
    for i in range(N-1):
        start = i * lbatch
        end = (i + 1) * lbatch
        # print(i, start, end)
        batches.append(encoded[start:end])
    batches.append(encoded[end:])
    # Decode
    ret = bytes()
    with fut.ThreadPoolExecutor(max_workers=N) as executor:
        # Submit tasks for execution and put pieces together
        for result  in executor.map(a2b_base64, batches):
            ret = ret + result
    return ret


def mutiproc_impl(encoded, N):
    """Try multi processing calling the underlying library"""
    # split the string into pieces
    d = len(encoded) // N            # number of splits
    lbatch = (d // 4) * 4           # lenght of first N-1 batches, the last is len(source) - lbatch*N
    batches = []
    for i in range(N-1):
        start = i * lbatch
        end = (i + 1) * lbatch
        # print(i, start, end)
        batches.append(encoded[start:end])
    batches.append(encoded[end:])
    # Decode
    ret = bytes()
    with fut.ProcessPoolExecutor(max_workers=N) as executor:
        # Submit tasks for execution and put pieces together
        for result  in executor.map(a2b_base64, batches):
            ret = ret + result
    return ret

if __name__ == "__main__":
    CPU_NUM = cpu_count()

    # Prepare a 4.6 GB byte string (with less than 32 GB ram you may experience swapping on virtual memory)
    repeat = 60000000
    large_b64_string = b'VGhpcyBzdHJpbmcgaXMgZm9ybWF0dGVkIHRvIGJlIGVuY29kZWQgd2l0aG91dCBwYWRkaW5nIGJ5dGVz' * repeat

    # Compare implementations
    print("\nreference_implementation")
    t_start = perf_counter()
    dec1 = reference_implementation(large_b64_string)
    t_end = perf_counter()
    print('decoding time =', (t_end - t_start))

    sleep(1)

    print("\nimplmementation1")
    t_start = perf_counter()
    dec2 = implmementation1(large_b64_string)
    t_end = perf_counter()
    print("Verify result", "Ok" if dec2==dec1 else "FAIL")
    print('decoding time =', (t_end - t_start))
    del dec2; collect()     # force freeing memory to avoid swapping on virtual mem

    sleep(1)

    print("\nthreaded_impl")
    t_start = perf_counter()
    dec3 = threaded_impl(large_b64_string, CPU_NUM)
    t_end = perf_counter()
    print("Verify result", "Ok" if dec3==dec1 else "FAIL")
    print('decoding time =', (t_end - t_start))
    del dec3; collect()

    sleep(1)

    print("\nmutiproc_impl")
    t_start = perf_counter()
    dec4 = mutiproc_impl(large_b64_string, CPU_NUM)
    t_end = perf_counter()
    print("Verify result", "Ok" if dec4==dec1 else "FAIL")
    print('decoding time =', (t_end - t_start))
    del dec4; collect()

问题原因分析 #

1. 多线程受GIL限制，无法真正并行 #

Python的全局解释器锁（GIL）会限制同一时刻只有一个线程执行Python字节码。Base64解码是CPU密集型任务，多线程不仅无法利用多核并行，还会因为线程切换产生额外开销，导致整体速度变慢。

2. 多进程的内存拷贝开销巨大 #

ProcessPoolExecutor在传递任务数据时，会通过序列化（pickle）把每个batch的字符串拷贝到子进程空间。对于GB级别的数据，这种跨进程的内存拷贝成本极高，完全抵消了并行解码带来的性能收益。

3. 低效的结果拼接方式 #

代码中用ret = ret + result拼接解码结果，每次拼接都会创建新的bytes对象并拷贝原有数据。对于GB级别的结果，这种操作会产生大量内存分配和拷贝开销，进一步拖慢速度。

4. 基准实现本身已是最优 #

base64.b64decode和a2b_base64都是基于C语言实现的高效函数，单线程执行时已经接近CPU的处理极限，并行方案的额外开销远大于其带来的收益。

优化建议 #

• 优先使用单线程基准实现：既然底层已经是C级别的高效实现，单线程就是最优选择，无需强行并行。
• 优化多进程方案（若必须并行）：
  • 使用共享内存（如multiprocessing.Array或mmap）传递Base64数据，避免跨进程拷贝。
  • 改用列表收集解码结果，最后用b''.join(results)拼接，大幅降低内存开销。
• 考虑流式处理：如果数据来自文件或网络，可以分块读取、分块解码，无需一次性加载全部数据到内存，同时能利用IO等待时间做并行处理，但纯内存数据场景下收益有限。

内容的提问来源于stack exchange，提问作者user27243451