Intel MPI 2021.1在RHEL8.6(KeyarchOS)上整数除零报错求助

问题背景 #

将HPC集群计算节点从CentOS 7升级至RHEL 8.6发行版（KeyarchOS 5.8）后，发现部分使用Intel OneAPI 2021.1编译的软件通过mpirun运行时频繁失败，典型报错为整数除零导致的浮点异常。测试中仅约20%概率运行成功，但在CentOS 7环境下可100%正常运行。

报错信息 #

[cu345:1485183:0:1485183] Caught signal 8 (Floating point exception: integer divide by zero)
[cu345:1485184:0:1485184] Caught signal 8 (Floating point exception: integer divide by zero)
[cu345:1485185:0:1485185] Caught signal 8 (Floating point exception: integer divide by zero)
[cu345:1485186:0:1485186] Caught signal 8 (Floating point exception: integer divide by zero)
==== backtrace (tid:1485126) ====
 0 0x0000000000012ce0 __funlockfile()  :0
 1 0x0000000000b696ed next_random()  /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/shm/posix/eager/include/intel_transport_types.h:1809
 2 0x0000000000b696ed impi_bcast_intra_huge()  /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/shm/posix/eager/include/intel_transport_bcast.h:667
 3 0x0000000000b6630d impi_bcast_intra_heap()  /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/shm/posix/eager/include/intel_transport_bcast.h:798
 4 0x000000000018ef6d MPIDI_POSIX_mpi_bcast()  /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/shm/src/../src/../posix/intel/posix_coll.h:124
 5 0x000000000017335e MPIDI_SHM_mpi_bcast()  /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/shm/src/../src/shm_coll.h:39
 6 0x000000000017335e MPIDI_Bcast_intra_composition_alpha()  /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_impl.h:303
 7 0x000000000017335e MPID_Bcast_invoke()  /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_select_utils.c:1726
 8 0x000000000017335e MPIDI_coll_invoke()  /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_select_utils.c:3356
 9 0x0000000000153bee MPIDI_coll_select()  /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_globals_default.c:129
10 0x000000000021c02d MPID_Bcast()  /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll.h:51
11 0x00000000001386e9 PMPI_Bcast()  /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/bcast/bcast.c:416

复现步骤 #

1. 编译LAMMPS #

module load intel/oneapi2021.1
module load gsl/2.8
# 下载LAMMPS源码包
tar -xvf lammps.tar.gz
cd lammps
mkdir build
cd build

cmake  -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_Fortran_COMPILER=gfortran -D BUILD_MPI=yes -D BUILD_OMP=yes -D LAMMPS_MACHINE=mpi -D BUILD_SHARED_LIBS=yes -D CMAKE_INSTALL_PREFIX=/public/software/lammps/29Aug2024 -D CMAKE_INSTALL_LIBDIR=lib -D CMAKE_INSTALL_FULL_LIBDIR=/public/software/lammps/2024/lib -C ../cmake/presets/most.cmake -C ../cmake/presets/nolib.cmake -D GSL_INCLUDE_DIR=/public/software/gsl/2.8/include -D GSL_LIBRARY=/public/software/gsl/2.8/lib/libgsl.a -DGSL_CBLAS_LIBRARY=/public/software/gsl/2.8/lib/libgslcblas.a -D PKG_PLUMED=yes -D FFT=MKL -D PKG_ELECTRODE=yes -D PKG_KSPACE=yes -D USE_INTERNAL_LINALG=yes ../cmake

make -j 32
make install

2. 提交运行脚本 #

使用以下SBATCH脚本提交任务：

#!/bin/bash -l
#SBATCH -p cpu
#SBATCH --qos normal

#SBATCH --nodes=1
#SBATCH --ntasks-per-node=64
#SBATCH -J std

module load intel/oneapi2021.1
module load gsl/2.8
module load lammps/2024.8.29

export I_MPI_DEBUG=6
mpirun lmp_mpi -i input.lammps && echo "mpi_success $SLURM_NODELIST" || echo "mpi_fail $SLURM_NODELIST"

问题原因 #

从报错栈可以定位到异常发生在Intel MPI的next_random()函数中，核心原因是Intel OneAPI 2021.1版本的MPI组件与RHEL 8.6（KeyarchOS 5.8）的高版本glibc（2.28）存在兼容性bug：

• CentOS 7使用的glibc版本为2.17，而RHEL 8.6升级到了glibc 2.28，Intel MPI 2021.1的共享内存通信模块中，随机数初始化逻辑在高版本glibc下存在竞态条件，多进程并发时随机数种子未正确初始化，触发整数除零错误。
• 概率性成功是因为竞态条件的随机性，偶尔会避开错误分支。

解决方案 #

1. 升级Intel OneAPI版本（推荐） #

Intel在2022.1及后续版本中修复了该兼容性问题，直接升级到最新稳定版的Intel OneAPI后，重新编译软件即可彻底解决问题。

2. 禁用Intel MPI的SHM随机数优化（临时方案） #

通过设置环境变量强制MPI使用兼容的随机数生成逻辑，无需重新编译软件：
修改SBATCH脚本，添加环境变量后再执行mpirun：

#!/bin/bash -l
#SBATCH -p cpu
#SBATCH --qos normal

#SBATCH --nodes=1
#SBATCH --ntasks-per-node=64
#SBATCH -J std

module load intel/oneapi2021.1
module load gsl/2.8
module load lammps/2024.8.29

export I_MPI_DEBUG=6
export I_MPI_SHM_RANDOM_GENERATOR=0  # 添加该行禁用SHM随机数优化
mpirun lmp_mpi -i input.lammps && echo "mpi_success $SLURM_NODELIST" || echo "mpi_fail $SLURM_NODELIST"

3. 回退glibc版本（不推荐） #

如果无法升级OneAPI，可尝试在节点上安装兼容的glibc 2.17版本，通过LD_LIBRARY_PATH指定优先加载该版本的glibc，但此操作会带来系统兼容性风险，仅作为临时应急方案。

内容的提问来源于stack exchange，提问作者link89