OpenMPI 4.1.2 基于InfiniBand Pkey网络运行挂起问题求助

环境与操作说明 #

• OpenMPI版本：4.1.2
• 执行命令：

mpiexec -n 8 -host cn0001-pkey,cn0002-pkey /home/ring_c/ring_c

• 网络配置：

cn0001
ib1 (默认pkey) 10.1.0.101
ib1.00a0 (pkey 0xa0) 10.2.0.101

cn0002
ib1 (默认pkey) 10.1.0.102
ib1.00a0 (pkey 0xa0) 10.2.0.102

• /etc/hosts配置：

10.2.0.101  cn0001-pkey
10.2.0.102  cn0002-pkey

问题现象 #

程序在cn0001上正常执行，但在cn0002上挂起，怀疑未通过Pkey网络通信。启用-mca plm_base_verbose 99后的日志如下：

mpiexec -n 8 -host cn0001-pkey,cn0002-pkey -mca plm_base_verbose 99 /home/ring_c/ring_c
[cn0001:05516] mca: base: components_register: registering framework plm components
[cn0001:05516] mca: base: components_register: found loaded component isolated
[cn0001:05516] mca: base: components_register: component isolated has no register or open function
[cn0001:05516] mca: base: components_register: found loaded component rsh
[cn0001:05516] mca: base: components_register: component rsh register function successful
[cn0001:05516] mca: base: components_register: found loaded component slurm
[cn0001:05516] mca: base: components_register: component slurm register function successful
[cn0001:05516] mca: base: components_open: opening plm components
[cn0001:05516] mca: base: components_open: found loaded component isolated
[cn0001:05516] mca: base: components_open: component isolated open function successful
[cn0001:05516] mca: base: components_open: found loaded component rsh
[cn0001:05516] mca: base: components_open: component rsh open function successful
[cn0001:05516] mca: base: components_open: found loaded component slurm
[cn0001:05516] mca: base: components_open: component slurm open function successful
[cn0001:05516] mca:base:select: Auto-selecting plm components
[cn0001:05516] mca:base:select:(  plm) Querying component [isolated]
[cn0001:05516] mca:base:select:(  plm) Query of component [isolated] set priority to 0
[cn0001:05516] mca:base:select:(  plm) Querying component [rsh]
[cn0001:05516] mca:base:select:(  plm) Query of component [rsh] set priority to 10
[cn0001:05516] mca:base:select:(  plm) Querying component [slurm]
[cn0001:05516] mca:base:select:(  plm) Selected component [rsh]
[cn0001:05516] mca: base: close: component isolated closed
[cn0001:05516] mca: base: close: unloading component isolated
[cn0001:05516] mca: base: close: component slurm closed
[cn0001:05516] mca: base: close: unloading component slurm
[cn0001:05516] [[33746,0],0] plm:rsh: final template argv:
        /usr/bin/ssh <template> ( test ! -r ./.profile || . ./.profile;           PATH=/opt/modules/openmpi/gcc/4.1.2/bin:$PATH ; export PATH ; LD_LIBRARY_PATH=/opt/modules/openmpi/gcc/4.1.2/lib:${LD_LIBRARY_PATH:-} ; export LD_LIBRARY_PATH ; DYLD_LIBRARY_PATH=/opt/modules/openmpi/gcc/4.1.2/lib:${DYLD_LIBRARY_PATH:-} ; export DYLD_LIBRARY_PATH ;   /opt/modules/openmpi/gcc/4.1.2/bin/orted -mca ess "env" -mca ess_base_jobid "2211577856" -mca ess_base_vpid "<template>" -mca ess_base_num_procs "2" -mca orte_node_regex "cn[4:1],cn[4:2]-pkey@0(2)" -mca orte_hnp_uri "2211577856.0;tcp://16.1.15.2,10.1.0.201,10.2.0.101:38393" -mca plm_base_verbose "99" -mca plm "rsh" --tree-spawn -mca routed "radix" -mca orte_parent_uri "2211577856.0;tcp://16.1.15.2,10.1.0.201,10.2.0.101:38393" -mca pmix "^s1,s2,cray,isolated" )
[cn0002:05935] mca: base: components_register: registering framework plm components
[cn0002:05935] mca: base: components_register: found loaded component rsh
[cn0002:05935] mca: base: components_register: component rsh register function successful
[cn0002:05935] mca: base: components_open: opening plm components
[cn0002:05935] mca: base: components_open: found loaded component rsh
[cn0002:05935] mca: base: components_open: component rsh open function successful
[cn0002:05935] mca:base:select: Auto-selecting plm components
[cn0002:05935] mca:base:select:(  plm) Querying component [rsh]
[cn0002:05935] mca:base:select:(  plm) Query of component [rsh] set priority to 10
[cn0002:05935] mca:base:select:(  plm) Selected component [rsh]

问题分析与解决方案 #

核心问题点 #

从日志的orte_hnp_uri字段可见，HNP（主机节点进程）暴露的地址包含非目标网段（如默认Pkey的10.1.0.201、16.1.15.2），cn0002上的orted进程可能尝试通过这些网段连接导致阻塞。另外，OpenMPI不会自动绑定指定Pkey接口，需显式配置。

解决步骤 #

1. 强制限制TCP通信使用目标Pkey网段
   在mpiexec命令中添加MCA参数，锁定通信仅使用10.2.0.0/24网段：

   mpiexec -n 8 -host cn0001-pkey,cn0002-pkey \
     -mca btl_tcp_if_include 10.2.0.0/24 \
     -mca oob_tcp_if_include 10.2.0.0/24 \
     /home/ring_c/ring_c
   

   • btl_tcp_if_include：限制字节传输层（BTL）的TCP接口范围
   • oob_tcp_if_include：限制带外通信（OOB）的TCP接口范围

2. 直接绑定InfiniBand设备与Pkey
   若使用原生InfiniBand BTL组件，需指定对应设备和Pkey：

   mpiexec -n 8 -host cn0001-pkey,cn0002-pkey \
     -mca btl_openib_include_devices ib1.00a0 \
     -mca btl_openib_pkey 0xa0 \
     /home/ring_c/ring_c
   

   • 先执行ompi_info | grep btl_openib确认OpenMPI已编译该组件

3. 验证SSH与主机解析

   • 确保两台机器的/etc/hosts都配置了cn0001-pkey和cn0002-pkey，且能免密SSH互访
   • 测试ssh cn0002-pkey hostname，确认无卡顿、直接返回主机名

4. 检查Pkey网段连通性

   • 执行ping cn0001-pkey和ping cn0002-pkey验证TCP连通
   • 用ibping测试InfiniBand链路：

     ibping -G 0xa0 -d ib1 -S 10.2.0.101 10.2.0.102
     

内容的提问来源于stack exchange，提问作者Astro.Bacon