MPI4PY跨节点Spawn子进程失败问题求助

在HPC集群上开发了一套基于MPI的程序：主进程为Python代码，通过mpi4py的MPI.COMM_SELF.Spawn启动N个运行Fortran可执行文件child的子进程，并使用MPI广播将字符串CNTRL发送给Fortran子进程。

集群每个节点含48核，当Spawn的子进程仅在单个节点内时，代码运行完全正常；但当申请跨节点的核心资源（例如通过SLURM申请2节点共65核，Spawn64个子进程）时，代码触发MPI_ERR_SPAWN错误，提示系统可用slot不足，无法Spawn进程。

相关代码 #

main.py #

from mpi4py import MPI
from env import myclass

# Define message to send to Python
req = b'CNTRL'
# Define class locally
C = myclass(req)
# Call class function to perform communication
C.myfunc()

print('Python done')

env.py #

from mpi4py import MPI

class myclass:
  def __init__(self,req):

    # sub_comm is an MPI intercommunicator
    self.sub_comm = MPI.COMM_SELF.Spawn('./child', args=[], maxprocs=32)
    # common_comm is an intracommunicator accross the python process and the spawned process.
    self.common_comm=self.sub_comm.Merge(False)
    self.size = self.common_comm.Get_size()
    self.rank = self.common_comm.Get_rank()
    
    # define message to send to fortran
    self.req = req 

  def myfunc(self):
    # Just to try, send a message
    self.common_comm.Bcast([self.req,  MPI.CHAR],   root=0)
    print('Python sent a broadcast message: ',self.req)
    self.closeMPI()

  def closeMPI(self):
    # free the (merged) intra communicator
    self.common_comm.Free()
    # disconnect the inter communicator is required to finalize the spawned process.
    self.sub_comm.Disconnect()

Fortran child程序 #

program child
  !
  use mpi
  !
  implicit none
  !
  integer :: parentcomm,intracomm,group,newgroup,newcomm
  integer :: rank,rank2,size,ierr,i
  character(len=5) :: request
  integer, allocatable, dimension(:) :: ranks
  !
  call MPI_INIT(ierr)
  call MPI_COMM_GET_PARENT(parentcomm, ierr)
  call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr)
  call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierr)
  call MPI_COMM_GROUP(MPI_COMM_WORLD, group, ierr)
  allocate(ranks(size))
  do i=1,size
    ranks(i) = i-1
  end do
  call MPI_GROUP_INCL(group,size,ranks,newgroup,ierr)
  call MPI_GROUP_SIZE(newgroup, size, ierr)
  call MPI_COMM_CREATE(MPI_COMM_WORLD, newgroup, newcomm, ierr)
  call MPI_INTERCOMM_MERGE(parentcomm, .true., intracomm, ierr)
  call MPI_COMM_RANK(intracomm, rank2, ierr)
  call MPI_COMM_SIZE(intracomm, size, ierr)

  call MPI_BCAST(request,5,MPI_CHAR,0,intracomm,ierr)
  if (rank == 0) print*, 'Message received in fortran: ',request

  if (rank == 1) print*, 'Child frees intracomm'
  call MPI_COMM_FREE(intracomm, ierr)
  if (rank == 1) print*, 'Child disconnects intercomm'
  call MPI_COMM_DISCONNECT(parentcomm, ierr)
  if (rank == 1) print*, 'Child finalises'
  call MPI_FINALIZE(ierr)
end program child

已尝试方案 #

• 启动MPI进程时使用use-hwthread-cpus和bind-to-core参数
• 从OpenMPI切换至MPICH，同步编译Fortran代码并重新安装mpi4py
• 在MPI调用中指定节点列表

当前环境：Python 3.10.12 + mpi4py 3.1.5，尝试过OpenMPI 4.1.1和MPICH 4.1.2版本。

1. 调整Spawn的通信域策略 #

MPI.COMM_SELF.Spawn默认仅在主进程所在节点分配资源，跨节点场景下需改用全局通信域：
修改env.py中的Spawn调用：

# 替换原COMM_SELF.Spawn为COMM_WORLD.Spawn
self.sub_comm = MPI.COMM_WORLD.Spawn('./child', args=[], maxprocs=32)

注意：启动主进程时需通过mpirun -n 1 python main.py将其加入全局MPI通信域，而非直接用python main.py启动。

2. 利用SLURM环境变量传递跨节点资源信息 #

通过SLURM提供的环境变量让MPI知晓整个作业的可用节点，修改env.py的Spawn逻辑：

import os
from mpi4py import MPI

class myclass:
  def __init__(self,req):
    info = MPI.Info.Create()
    # 获取SLURM节点列表并传递给Spawn
    nodelist = os.environ.get('SLURM_JOB_NODELIST')
    if nodelist:
        # OpenMPI用host参数，MPICH用nodes参数，根据实际环境调整
        info.Set('host', nodelist)
    # Spawn时传入资源信息
    self.sub_comm = MPI.COMM_SELF.Spawn('./child', args=[], maxprocs=32, info=info)
    # 后续原有逻辑不变...

3. 允许MPI进程超配（针对OpenMPI） #

在启动主进程时添加--oversubscribe参数，允许MPI在已分配的slot内Spawn子进程：

mpirun -n 1 --oversubscribe python main.py

需确保作业申请的总核数≥主进程数+Spawn子进程数（比如申请65核对应1主+64子）。

4. 简化Fortran子进程的通信域操作 #

原代码中MPI_GROUP_INCL和MPI_COMM_CREATE属于冗余操作，会增加通信域冲突风险，修改后的Fortran代码：

program child
  use mpi
  implicit none
  integer :: parentcomm,intracomm
  integer :: rank,size,ierr
  character(len=5) :: request

  call MPI_INIT(ierr)
  call MPI_COMM_GET_PARENT(parentcomm, ierr)
  call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr)
  call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierr)

  call MPI_INTERCOMM_MERGE(parentcomm, .true., intracomm, ierr)
  call MPI_BCAST(request,5,MPI_CHAR,0,intracomm,ierr)
  if (rank == 0) print*, 'Message received in fortran: ',request

  call MPI_COMM_FREE(intracomm, ierr)
  call MPI_COMM_DISCONNECT(parentcomm, ierr)
  call MPI_FINALIZE(ierr)
end program child

内容的提问来源于stack exchange，提问作者Giorgio Maria Cavallazzi