为何pbdMPI中指定comm.rank执行合并的for循环失效？

我用pbdMPI并行化R脚本，10个comm.rank各加载1个文件，之后由comm.rank 0收集所有文件并执行合并。但写的for循环没生效，所有rank都尝试合并，导致待合并项耗尽时脚本停滞。

脚本流程：10个基因矩阵（每个rank加载1个）先标准化，rank 0把每个Seurat对象收集到列表x，将x第一个元素赋值给Seuratdata，再让rank 0合并x中剩余对象。

相关代码 #

x<-gather(Obj, rank.dest=0)
rm(Obj)
comm.print(x, all.rank=TRUE)
invisible(gc())
barrier()

comm.print("starting merging")
#comm.print(x, all.rank=TRUE)

comm.print(paste("length of x is", length(x)), all.rank=TRUE)
if(comm.rank()==0){
        x<-x
    Seuratdata<-x[[1]]

} else {
x<-NULL
}

if(comm.rank()==0){
        for(i in 2:length(x)){
        comm.print(paste("merging object", i),all.rank=TRUE)
        comm.print(i)
        comm.print(x[[i]])
    Seuratdata<-merge(Seuratdata, x[[i]], add.cell.ids=NULL, merge.data=TRUE)
    x[[i]]<-NA
        comm.print(paste("done merging", i))
}
}

输出信息 #

comm.print(x, all.rank=TRUE)的输出： #

COMM.RANK = 0
[[1]]
An object of class Seurat 
114 features across 4034 samples within 1 assay 
Active assay: RNA (114 features, 0 variable features)

[[2]]
An object of class Seurat 
114 features across 1690 samples within 1 assay 
Active assay: RNA (114 features, 0 variable features)
..........

[[9]]
An object of class Seurat 
114 features across 3137 samples within 1 assay 
Active assay: RNA (114 features, 0 variable features)

[[10]]
An object of class Seurat 
114 features across 3601 samples within 1 assay 
Active assay: RNA (114 features, 0 variable features)

COMM.RANK = 1
NULL
COMM.RANK = 2
NULL
COMM.RANK = 3
NULL
COMM.RANK = 4
NULL
COMM.RANK = 5
NULL
COMM.RANK = 6
NULL
COMM.RANK = 7
NULL
COMM.RANK = 8
NULL
COMM.RANK = 9
NULL

comm.print(i, all.rank=TRUE)的输出： #

COMM.RANK = 0
[1] 2
COMM.RANK = 1
[1] 2
COMM.RANK = 2
[1] 2
COMM.RANK = 3
[1] 2
COMM.RANK = 4
[1] 2
COMM.RANK = 5
[1] 2
COMM.RANK = 6
[1] 2
COMM.RANK = 7
[1] 2
COMM.RANK = 8
[1] 2
COMM.RANK = 9
[1] 2
An object of class Seurat 
114 features across 1690 samples within 1 assay 
Active assay: RNA (114 features, 0 variable features)
COMM.RANK = 0
[1] 3
COMM.RANK = 1
[1] 1
COMM.RANK = 2
[1] 1
COMM.RANK = 3
[1] 1
COMM.RANK = 4
[1] 1
COMM.RANK = 5
[1] 1
COMM.RANK = 6
[1] 1
COMM.RANK = 7
[1] 1
COMM.RANK = 8
[1] 1
COMM.RANK = 9
[1] 1
An object of class Seurat 
114 features across 3989 samples within 1 assay 
Active assay: RNA (114 features, 0 variable features)
COMM.RANK = 0
[1] 4
COMM.RANK = 1
[1] 0
COMM.RANK = 2
[1] 0
COMM.RANK = 3
[1] 0
COMM.RANK = 4
[1] 0
COMM.RANK = 5
[1] 0
COMM.RANK = 6
[1] 0
COMM.RANK = 7
[1] 0
COMM.RANK = 8
[1] 0
COMM.RANK = 9
[1] 0
An object of class Seurat 
114 features across 2421 samples within 1 assay 
Active assay: RNA (114 features, 0 variable features)
COMM.RANK = 0
[1] 5
COMM.RANK = 1
[1] 2
COMM.RANK = 2
[1] 2
COMM.RANK = 3
[1] 2
COMM.RANK = 4
[1] 2
COMM.RANK = 5
[1] 2
COMM.RANK = 6
[1] 2
COMM.RANK = 7
[1] 2
COMM.RANK = 8
[1] 2
COMM.RANK = 9
[1] 2
An object of class Seurat 
114 features across 4402 samples within 1 assay 
Active assay: RNA (114 features, 0 variable features)
COMM.RANK = 0
[1] 6
COMM.RANK = 1
[1] 1
COMM.RANK = 2
[1] 1
COMM.RANK = 3
[1] 1
COMM.RANK = 4
[1] 1
COMM.RANK = 5
[1] 1
COMM.RANK = 6
[1] 1
COMM.RANK = 7
[1] 1
COMM.RANK = 8
[1] 1
COMM.RANK = 9
[1] 1
An object of class Seurat 
114 features across 3480 samples within 1 assay 
Active assay: RNA (114 features, 0 variable features)
COMM.RANK = 0
[1] 7
COMM.RANK = 1
[1] 0
COMM.RANK = 2
[1] 0
COMM.RANK = 3
[1] 0
COMM.RANK = 4
[1] 0
COMM.RANK = 5
[1] 0
COMM.RANK = 6
[1] 0
COMM.RANK = 7
[1] 0
COMM.RANK = 8
[1] 0
COMM.RANK = 9
[1] 0

解决方案 #

问题出在非0 rank的变量作用域和逻辑执行范围：虽然给非0 rank的x赋值为NULL，但循环内的comm.print(i)等语句未完全包裹在if(comm.rank()==0)中，导致所有rank都会执行这些操作；非0 rank中x为NULL，length(x)返回0，循环for(i in 2:0)生成无效序列，引发异常导致脚本停滞。

修改后的代码：

x <- gather(Obj, rank.dest=0)
rm(Obj)
comm.print(x, all.rank=TRUE)
invisible(gc())
barrier()

comm.print("starting merging")
comm.print(paste("length of x is", length(x)), all.rank=TRUE)

# 仅rank0处理合并逻辑，其他rank直接跳过
if(comm.rank() == 0){
    Seuratdata <- x[[1]]
    # 循环合并剩余对象
    for(i in 2:length(x)){
        comm.print(paste("merging object", i), all.rank=TRUE)
        comm.print(i, rank.dest=0)  # 仅rank0打印i，避免输出混乱
        comm.print(x[[i]], rank.dest=0)
        Seuratdata <- merge(Seuratdata, x[[i]], add.cell.ids=NULL, merge.data=TRUE)
        x[[i]] <- NA
        comm.print(paste("done merging", i), all.rank=TRUE)
    }
} else {
    # 非0 rank清空x，释放内存
    x <- NULL
}

# 确保所有rank同步后再继续后续操作
barrier()

关键调整说明：

1. 把Seuratdata初始化、合并循环及内部所有操作完全放在if(comm.rank()==0)代码块内，非0 rank不会进入该逻辑。
2. 对不需要全局打印的内容（如i和x[[i]]），使用rank.dest=0限定仅rank0输出，避免无效信息干扰。
3. 非0 rank的x赋值NULL放在else分支，逻辑更清晰。
4. 添加最后一个barrier()确保所有rank在合并完成后同步，避免后续操作不同步。

内容的提问来源于stack exchange，提问作者mahnoorh