Ubuntu环境下mpi4py与OpenMPI突然不兼容问题求助

问题背景 #

环境：Ubuntu 20.04、Python 3.8、mpi4py 3.1.5、OpenMPI 4.1.4
无任何系统或包更新操作的情况下，所有加载mpi4py的Python代码均报错，但OpenMPI单独测试（mpirun -np 4 hostname）正常。

错误日志 #

[juanMS:15643] [[INVALID],INVALID] ORTE_ERROR_LOG: A system-required executable either could not be found or was not executable by this user in file ess_singleton_module.c at line 572
[juanMS:15643] [[INVALID],INVALID] ORTE_ERROR_LOG: A system-required executable either could not be found or was not executable by this user in file ess_singleton_module.c at line 172
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  orte_ess_init failed
  --> Returned value A system-required executable either could not be found or was not executable by this user (-126) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  ompi_mpi_init: ompi_rte_init failed
  --> Returned "A system-required executable either could not be found or was not executable by this user" (-126) instead of "Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[juanMS:15643] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!

已尝试操作 #

• 卸载系统及Python虚拟环境中所有OpenMPI相关包：

  sudo apt purge --autoremove libopenmpi-dev libopenmpi3 mpich openmpi-bin openmpi-common
  pip uninstall mpi4py
  

• 多次重复上述卸载操作，错误仍未解决

可复现脚本 #

from mpi4py import MPI
import sys

def print_hello(rank, size, name):
    msg = "Hello World! I am process {0} of {1} on {2}.\n"
    sys.stdout.write(msg.format(rank, size, name))

if __name__ == "__main__":
    size = MPI.COMM_WORLD.Get_size()
    rank = MPI.COMM_WORLD.Get_rank()
    name = MPI.Get_processor_name()

    print_hello(rank, size, name)

补充测试结果 #

mpi4py=3.1.5与openmpi=4.0.X及mpich=3.3.2兼容。

解决步骤 #

1. 检查并修复OpenMPI环境变量 #

执行以下命令查看OpenMPI相关路径是否在环境变量中：

echo $PATH | grep openmpi
echo $LD_LIBRARY_PATH | grep openmpi

若输出为空或路径错误，手动添加正确路径（以系统默认安装路径为例）：

export PATH=/usr/lib/openmpi/bin:$PATH
export LD_LIBRARY_PATH=/usr/lib/openmpi/lib:$LD_LIBRARY_PATH

如需永久生效，将上述两行添加到~/.bashrc或虚拟环境的activate脚本中。

2. 重新编译绑定当前OpenMPI的mpi4py #

卸载现有mpi4py后，通过编译方式安装，确保与当前OpenMPI版本绑定：

# 安装OpenMPI开发依赖
sudo apt install libopenmpi-dev openmpi-bin
# 在虚拟环境中执行编译安装
pip install mpi4py --no-binary :all: --install-option="--mpi=openmpi"

3. 检查OpenMPI可执行文件权限 #

针对错误提示的“可执行文件权限”问题，检查并修复权限：

ls -l /usr/bin/orte* /usr/bin/ompi*
# 若权限不足，执行：
sudo chmod +x /usr/bin/orte* /usr/bin/ompi*

4. 清理残留缓存与配置文件 #

彻底清理卸载残留的文件：

# 清理系统包缓存
sudo apt clean && sudo apt autoclean
# 清理Python pip缓存
rm -rf ~/.cache/pip
# 删除OpenMPI残留目录（如有）
sudo rm -rf /usr/local/lib/openmpi /usr/local/bin/openmpi

5. 降级到兼容的OpenMPI版本 #

利用已验证的兼容版本组合，降级OpenMPI：

# 卸载当前OpenMPI
sudo apt purge --autoremove openmpi-bin libopenmpi-dev
# 安装兼容的4.0.X版本
sudo apt install openmpi-bin=4.0.3-0ubuntu1 libopenmpi-dev=4.0.3-0ubuntu1
# 重新安装mpi4py
pip install mpi4py==3.1.5

内容的提问来源于stack exchange，提问作者paeion