Hey there! I get why this is confusing—multiprocessing in Python can trip you up because each process has its own isolated memory space. When you pass a regular array to a child process, it makes a copy, so any writes you do won't show up in the parent. Let's fix that with shared memory approaches tailored to your block-processing use case.

1. Basic Shared Arrays with multiprocessing.Array #

This is the go-to for simple, typed arrays (integers, floats, etc.). It creates a shared memory block that all processes can access directly.

Example Code: #

import multiprocessing

def process_block(shared_arr, start_idx, end_idx):
    # The shared array acts like a list for direct indexing
    for i in range(start_idx, end_idx):
        # Replace this with your actual processing logic
        arr[i] = arr[i] * 2  # Example: double each element in the block

if __name__ == "__main__":
    # Define your giant array parameters
    total_size = 10_000_000
    block_size = 2_000_000

    # Create a shared array (type 'i' for int, 'd' for double/float)
    shared_arr = multiprocessing.Array('i', total_size)

    # Initialize the array (adjust based on your raw data)
    for i in range(total_size):
        shared_arr[i] = i

    # Split into blocks and spawn processes
    processes = []
    for start in range(0, total_size, block_size):
        end = min(start + block_size, total_size)
        p = multiprocessing.Process(target=process_block, args=(shared_arr, start, end))
        processes.append(p)
        p.start()

    # Wait for all processes to finish
    for p in processes:
        p.join()

    # Verify results
    print(shared_arr[:10])  # Should output [0,2,4,...,18]

Key Notes: #

• The type code (like 'i') defines the array's data type—check Python's array module docs for all valid options.
• Since each process only modifies its assigned block, no locks are needed (no overlapping access to the same indices).
• The shared array is backed by shared memory, so no copies are made when passing it to processes.

2. For NumPy Arrays: Use multiprocessing.shared_memory #

If you're working with NumPy arrays (way more efficient for large numerical datasets), the shared_memory module lets you create a shared memory segment that multiple processes can map to their own NumPy arrays—no data copying involved!

Example Code: #

import multiprocessing
import numpy as np
from multiprocessing import shared_memory

def process_numpy_block(shm_name, shape, dtype, start_idx, end_idx):
    # Attach to the existing shared memory block
    existing_shm = shared_memory.SharedMemory(name=shm_name)
    # Map the shared memory to a NumPy array
    arr = np.ndarray(shape, dtype=dtype, buffer=existing_shm.buf)
    
    # Process your target block
    arr[start_idx:end_idx] = arr[start_idx:end_idx] * 2  # Example processing
    
    # Clean up: close the shared memory (parent handles unlinking)
    existing_shm.close()

if __name__ == "__main__":
    total_size = 10_000_000
    block_size = 2_000_000
    dtype = np.int32

    # Create your base NumPy array
    giant_arr = np.arange(total_size, dtype=dtype)
    
    # Create shared memory and copy the array into it
    shm = shared_memory.SharedMemory(create=True, size=giant_arr.nbytes)
    shared_np_arr = np.ndarray(giant_arr.shape, dtype=giant_arr.dtype, buffer=shm.buf)
    shared_np_arr[:] = giant_arr[:]  # Copy data to shared memory

    # Spawn processes for each block
    processes = []
    for start in range(0, total_size, block_size):
        end = min(start + block_size, total_size)
        p = multiprocessing.Process(
            target=process_numpy_block,
            args=(shm.name, giant_arr.shape, dtype, start, end)
        )
        processes.append(p)
        p.start()

    # Wait for all processes to complete
    for p in processes:
        p.join()

    # Verify results
    print(shared_np_arr[:10])  # Should output [0,2,4,...,18]

    # Clean up shared memory to avoid system leaks
    shm.close()
    shm.unlink()

Key Notes: #

• This is ideal for giant arrays because it skips copying the entire dataset to each process.
• The parent creates the shared memory, and child processes only attach to it—minimizing overhead.
• Always unlink the shared memory when done, or it might linger in your system.

Common Pitfalls to Avoid #

• Never pass regular lists/NumPy arrays directly to processes: They get serialized and copied, so writes won't affect the parent's array. Stick to shared memory mechanisms.
• Double-check block boundaries: Ensure start/end indices don't overlap or go out of bounds to avoid errors.
• Match data types: Make sure the type code (for multiprocessing.Array) or dtype (for NumPy) is consistent across all processes—mismatches will cause garbage data.

Hope this clears things up! Let me know if you need help adapting this to your specific processing logic.

内容的提问来源于stack exchange，提问作者s6hebern