无法使用mpiexec/mpirun时，多主机运行mpi4py脚本的方案咨询

Absolutely! You can get your mpi4py script up and running across multiple hosts without relying on mpiexec or mpirun—you just need to tweak your script and use some MPI built-in features or custom discovery logic. Let’s break down the practical approaches:

1. Manual MPI Initialization with Command-Line Arguments #

MPI allows manual initialization where you explicitly define each process’s rank, total size, and peer host information. This bypasses the need for a launcher like mpiexec.

Modify your script to accept rank, total size, and master host as command-line args, then initialize MPI manually:

from mpi4py import MPI
import sys

# Validate input args
if len(sys.argv) != 4:
    print("Usage: python my_mpi_script.py <rank> <total_size> <master_host>")
    sys.exit(1)

rank = int(sys.argv[1])
total_size = int(sys.argv[2])
master_host = sys.argv[3]

# Initialize MPI with custom arguments
MPI.Init(sys.argv)

# Access the global communicator
comm = MPI.COMM_WORLD

# Verify setup
print(f"Rank {rank}/{total_size} running on host: {MPI.Get_processor_name()}")

# --- Your existing script logic goes here ---

MPI.Finalize()

How to run: #

• On your master node (rank 0):
  python my_mpi_script.py 0 8 master.yourdomain.com
• On each worker node (ranks 1 to 7):
  python my_mpi_script.py 1 8 master.yourdomain.com
  (Replace 1 with the corresponding rank for each worker)

2. Use MPI Environment Variables for Auto-Discovery #

Many MPI implementations (like OpenMPI) support configuring the runtime via environment variables, eliminating the need for mpiexec. This works if your system already runs the same command across hosts.

For OpenMPI: #

Set these environment variables on each host before running your script:

# Common settings for all hosts
export OMPI_MCA_orte_ess=env
export OMPI_MCA_orte_node_list="host1,host2,host3,host4" # List ALL your hosts here
export OMPI_MCA_orte_size=8 # Total number of processes

# On rank 0 host
export OMPI_MCA_orte_rank=0
python my_mpi_script.py

# On rank 1 host
export OMPI_MCA_orte_rank=1
python my_mpi_script.py

# Repeat for ranks 2-7

Note: Environment variable names may vary for other MPI implementations (e.g., MPICH uses different vars), so check your MPI docs for specifics.

3. Custom TCP-Based Process Discovery #

If you need fully automatic peer discovery (no manual rank/env setup), you can implement a simple TCP handshake to let processes find each other, then build an MPI communicator across hosts.

Here’s a simplified example:

from mpi4py import MPI
import socket

# Configure master host and port (adjust to your setup)
MASTER_HOST = "master.yourdomain.com"
MASTER_PORT = 12345

# Initialize MPI for the local process
MPI.Init()
self_comm = MPI.COMM_SELF

if self_comm.Get_rank() == 0:
    # Master process: listen for worker connections
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.bind((MASTER_HOST, MASTER_PORT))
        s.listen(7) # Wait for 7 workers
        
        intercomms = []
        for _ in range(7):
            conn, _ = s.accept()
            # Receive worker's MPI port string
            worker_port = conn.recv(1024).decode()
            # Connect to the worker's MPI communicator
            intercomm = self_comm.Connect(worker_port)
            intercomms.append(intercomm)
            conn.close()
    
    # Merge all inter-communicators into a global COMM_WORLD
    global_comm = MPI.COMM_WORLD.Merge(intercomms, root=0)
else:
    # Worker process: connect to master and send MPI port
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.connect((MASTER_HOST, MASTER_PORT))
        # Get local MPI port and send to master
        local_port = self_comm.Get_port()
        s.send(local_port.encode())
    
    # Wait for master to connect
    intercomm = self_comm.Accept(local_port)
    # Merge into global communicator
    global_comm = MPI.COMM_WORLD.Merge(intercomm, root=0)

# Now use global_comm for all your MPI operations
print(f"Global rank {global_comm.Get_rank()}/{global_comm.Get_size()} on {MPI.Get_processor_name()}")

# --- Your existing script logic goes here ---

MPI.Finalize()

How to run: #

1. Start the master process first on your designated master host:
   python my_mpi_script.py
2. Run the same command on all 7 worker hosts—they’ll automatically connect to the master and form a global MPI communicator.

Critical Notes #

• Ensure all hosts have matching MPI and mpi4py versions to avoid compatibility issues.
• Verify that firewall rules allow communication between hosts (both MPI’s dynamic ports and any custom TCP ports you use).
• For large clusters, consider adding timeout handling and error recovery to your custom discovery logic.

内容的提问来源于stack exchange，提问作者Andre