Python代码从Multiprocess迁移至MPI4Py的正确实现求助

Let's break down what's going wrong with your current mpi4py code and walk through a corrected, production-ready implementation for HPC environments.

Key Issues in Your Current Attempt #

• You forgot to import the MPI module from mpi4py—this is a critical missing piece that's causing early failures on non-rank-0 processes.
• You're scattering work items but never executing your run function on the received tasks.
• There's no handling for race conditions when creating output directories across multiple nodes.
• Your code uses Python 2 syntax (e.g., print name, iterkeys()) which may cause compatibility issues on modern HPC clusters.

Corrected mpi4py Implementation #

Here's the full adjusted code, with HPC-specific considerations built in:

import Home
import pickle
import time
import os
from mpi4py import MPI  # Critical MPI import

def run(a):
    name = a['name']
    people = a['people']
    save_path = 'res_semi_def'
    
    # Use a shared filesystem path for HPC (adjust this to your cluster's shared scratch space!)
    full_save_path = os.path.join(os.getcwd(), save_path)
    
    # Avoid race conditions when creating the output directory
    comm = MPI.COMM_WORLD
    comm.Barrier()
    if comm.rank == 0:
        os.makedirs(full_save_path, exist_ok=True)
    comm.Barrier()
    
    if sum(people.values()) > 0:
        start = time.time()
        home = Home.Home(people)
        try:
            home.simulate()
            print(f"{name} completed in {time.time()-start:.2f} [s]")
            fname = f"{name}.pkl"
            with open(os.path.join(full_save_path, fname), "wb") as f:
                pickle.dump(home.Consumption, f)
        except Exception as e:
            print(f"{name} error: {str(e)}")
            pass

def main():
    comm = MPI.COMM_WORLD
    rank = comm.rank
    size = comm.size

    # Only rank 0 loads the input data (avoids redundant file reads)
    if rank == 0:
        start_time = time.time()
        fname = 'censimento2011_full_par.pkl'
        # Ensure this file is on a shared filesystem accessible by all nodes!
        with open(fname, 'rb') as f:  # Use binary mode for Python 3 compatibility
            houses = pickle.load(f)
        # Prepare work items, pad with None to match process count for scatter
        work_items = [{'name': name, 'people': people} for name, people in houses.items()]
        pad_length = (size - len(work_items) % size) % size
        work_items += [None] * pad_length
    else:
        work_items = None
        start_time = None

    # Distribute work items to all processes
    my_work = comm.scatter(work_items, root=0)

    # Execute task if we have valid work
    if my_work is not None:
        run(my_work)

    # Wait for all processes to finish before finalizing
    comm.Barrier()

    # Only rank 0 prints total runtime
    if rank == 0:
        total_time = time.time() - start_time
        print(f"Total time: {total_time:.2f} [s]")

if __name__ == "__main__":
    main()

Critical HPC Deployment Notes #

• Shared Filesystem: Your input file (censimento2011_full_par.pkl) and output directory must live on a shared filesystem (like Lustre or NFS) so all compute nodes can access them. Local node storage won't work across multiple nodes.
• MPI Execution: To run this on your cluster, use a job scheduler command like srun or mpiexec:

  srun -N 2 -n 12 python your_script.py
  

  (This example uses 2 nodes with 6 processes each—adjust based on your cluster's resources.)
• Python Version: The code uses Python 3 syntax. If you're stuck on Python 2, revert to print name instead of f-strings and use iteritems() instead of items().
• Work Distribution: We pad work items with None to ensure scatter works evenly across processes. Processes will ignore these None entries.

Troubleshooting Your Original Error #

The exit codes (4, 7) and proxy errors you saw were almost certainly caused by:

1. Missing MPI import leading to failures on non-rank-0 processes.
2. No task execution after scattering work, causing processes to exit prematurely.
3. Potential file access issues if your input file wasn't on a shared filesystem.

内容的提问来源于stack exchange，提问作者Lorenzo Bottaccioli