使用SLURM提交R MPI任务时遇UCX地址接收失败错误求助

刚接触超级计算机与MPI，可访问带MPI分区的HPC集群，尝试通过SLURM提交多节点任务，使用的R脚本与批处理脚本如下，任务分配到节点后运行报错。

R脚本 #

library(snow) 
library(doSNOW) 
library(foreach)
library(Rmpi)
cl<- makeCluster(4-1, type="MPI")
registerDoSNOW(cl)
np<-getDoParWorkers()
np

lista = list()

foreach(i = 1:1000) %do%{
  lista[[i]] = matrix(rnorm(36),nrow=1000, ncol = 1000)
}

listainv = list()

foreach(i = 1:1000) %do%{
  listainv[[i]] = solve(lista[[i]])
}

listainv[[1]]

批处理脚本 #

#!/bin/bash -l
#SBATCH --job-name MyR
#SBATCH --partition=rome --mem=0 --time=1-24:00:00
#SBATCH --output=./Results/Results_from_jobs/final_b.out
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=2

MyRProgram="./test_scripts/sample_code_matrix.R"

module use /opt/pkg/ITER/modules/all
module load R

mpirun -np 1 R --oversubscribe --vanilla -f $MyRProgram

运行错误信息 #

> library(snow)
> library(doSNOW)
Loading required package: foreach
Loading required package: iterators
> library(foreach)
> library(Rmpi)
> cl<- makeCluster(4-1, type="MPI")
[cn49.hpc:19074] pml_ucx.c:178  Error: Failed to receive UCX worker address: Not found (-13)
[cn49.hpc:19074] [[24597,2],2] ORTE_ERROR_LOG: Error in file dpm/dpm.c at line 493
[cn49.hpc:19073] pml_ucx.c:178  Error: Failed to receive UCX worker address: Not found (-13)
[cn49.hpc:19073] [[24597,2],1] ORTE_ERROR_LOG: Error in file dpm/dpm.c at line 493
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  ompi_dpm_dyn_init() failed
  --> Returned "Error" (-1) instead of "Success" (0)
--------------------------------------------------------------------------
[cn49:19074] *** An error occurred in MPI_Init
[cn49:19074] *** reported by process [1611988994,2]
[cn49:19074] *** on a NULL communicator
[cn49:19074] *** Unknown error
[cn49:19074] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[cn49:19074] ***    and potentially your MPI job)
[cn48.hpc:38141] 1 more process has sent help message help-mpi-runtime.txt / mpi_init:startup:internal-failure
[cn48.hpc:38141] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[cn48.hpc:38141] 1 more process has sent help message help-mpi-errors.txt / mpi_errors_are_fatal unknown handle

• MPI启动逻辑冲突：批处理脚本用mpirun -np 1仅启动1个MPI进程，但SLURM已分配2节点×2核=4个任务槽；同时R脚本手动指定创建3个工作进程，这种混合启动方式导致UCX通信框架无法正确建立节点间连接，触发地址接收失败。
• 未启用并行执行：R脚本中使用%do%而非%dopar%，即使集群配置正确，也只会单线程运行，完全浪费并行资源。
• UCX动态进程管理冲突：部分HPC集群的Open MPI默认使用UCX通信框架，动态进程管理（DPM）模式下可能存在节点间通信配置不兼容，导致MPI初始化失败。

1. 修正批处理脚本的MPI启动方式 #

SLURM环境下无需手动指定进程数，让mpirun自动继承SLURM分配的任务数，同时确保加载匹配的MPI模块（Rmpi需要与系统MPI版本兼容）：

#!/bin/bash -l
#SBATCH --job-name MyR
#SBATCH --partition=rome --mem=0 --time=1-24:00:00
#SBATCH --output=./Results/Results_from_jobs/final_b.out
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=2

MyRProgram="./test_scripts/sample_code_matrix.R"

# 先加载MPI模块，再加载R，确保Rmpi链接正确的MPI库
module use /opt/pkg/ITER/modules/all
module load OpenMPI  # 根据集群实际MPI模块名称调整
module load R

# 去掉-np 1和--oversubscribe，让mpirun自动使用SLURM分配的任务数
mpirun R --vanilla -f $MyRProgram

2. 调整R脚本的并行逻辑 #

• 用Rmpi::mpi.universe.size()自动获取MPI总进程数，主进程除外即为工作进程数
• 将%do%替换为%dopar%启用并行执行
• 修正原脚本中矩阵生成的参数错误（rnorm(36)无法生成1000×1000矩阵所需的元素）
• 最后记得清理集群资源并终止MPI：

library(snow) 
library(doSNOW) 
library(foreach)
library(Rmpi)

# 自动获取可用MPI工作进程数（总进程数减1，排除主进程）
n_workers <- mpi.universe.size() - 1
cl <- makeCluster(n_workers, type="MPI")
registerDoSNOW(cl)
np <- getDoParWorkers()
cat("Number of workers:", np, "\n")

# 并行生成矩阵列表
lista <- foreach(i = 1:1000) %dopar% {
  matrix(rnorm(1000*1000), nrow=1000, ncol=1000)
}

# 并行求矩阵逆
listainv <- foreach(i = 1:1000) %dopar% {
  solve(lista[[i]])
}

print(listainv[[1]])

# 清理资源
stopCluster(cl)
mpi.quit()

3. 解决UCX通信问题（可选） #

如果仍遇到UCX相关错误，可在批处理脚本中添加环境变量禁用动态进程管理或切换通信层：

# 在mpirun前添加以下环境变量
export OMPI_MCA_orte_dpm=disabled
export OMPI_MCA_pml=ob1  # 切换到ob1通信层替代UCX

mpirun R --vanilla -f $MyRProgram

内容的提问来源于stack exchange，提问作者luciixi99