mpirun引发Python进程管理器内存泄漏致超算节点故障求助

在大学集群单个节点上运行Quantum Espresso的pw.x，通过Python脚本管理文件操作、进程调用和数据提取，核心代码如下：

types=["GaAs","InAs","GaSb","InSb"]
calcs=["scf","nscf","bands","pp"]
bandgaps=[]
for typ in types:
    for calc in calcs:
        shutil.copy("{0}/{1}.U.{2}.in".format(curdir,typ,calc),\
                    "{0}/{1}.int.{2}.in".format(intdir,typ,calc))
        filein1="{0}/{1}.int.{2}.in".format(intdir,typ,calc)
        command=['mpirun -n {2} pw.x < {0}.int.{1}.in &> {0}.int.{1}.out'.format(typ,calc,num_procs)]
        subprocess.run(command,cwd=intdir,shell=True)
    bandgaps.append(QE_extract_bandgap("{0}/{1}.int.nscf.out".format(intdir,typ)))
# 前期脚本负责向输入文件填充其他参数，此处为简化编辑

其中num_procs为处理器数量，typ对应材料类型，calc对应计算类型（scf、nscf、bands等）。

计算循环中出现可通过参数调整解决的常规错误：

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
     Error in routine mix_rho (1):
     negative dr2
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

     stopping ...

该错误出现后，基于scf的nscf和bands计算仍能继续，但处理下一个typ时，输出中出现大量Bus Error：

[tcad37.iue.tuwien.ac.at:mpi_rank_18][error_sighandler] Caught error: Bus error (signal 7)
[tcad37.iue.tuwien.ac.at:mpi_rank_19][error_sighandler] Caught error: Bus error (signal 7)
[tcad37.iue.tuwien.ac.at:mpi_rank_27][error_sighandler] Caught error: Bus error (signal 7)
[tcad37.iue.tuwien.ac.at:mpi_rank_34][error_sighandler] Caught error: Bus error (signal 7)
[tcad37.iue.tuwien.ac.at:mpi_rank_36][error_sighandler] Caught error: Bus error (signal 7)
[tcad37.iue.tuwien.ac.at:mpi_rank_0][error_sighandler] Caught error: Bus error (signal 7)

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 180018 RUNNING AT tcad37
=   EXIT CODE: 7
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Bus error (signal 7)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions

程序终止后，运行该任务的节点彻底无法使用，所有提交到该节点的任务均提示无可用内存。

使用的Slurm脚本如下：

#!/bin/sh
#SBATCH -J 3Uopt
#SBATCH --output=opt3.txt
#SBATCH -n 40
#SBATCH --ntasks-per-node=40
#SBATCH --time 3-00:00:00   # max runtime hh:mm:ss

推测问题根源是mpirun进程出错时未正常关闭shell，导致节点资源被占用却未正确释放，怀疑是subprocess.run与mpirun在处理崩溃进程时的兼容性问题。目前调试受限，失败会导致更多节点故障，恳请提供解决建议。

内容的提问来源于stack exchange，提问作者Don_Augusto