两台EC2实例上PyTorch分布式Gloo后端启动失败挂起

问题背景 #

两台EC2实例配置PyTorch分布式训练，rank0和rank1分别设置环境变量（含MASTER_ADDR、MASTER_PORT、GLOO_SOCKET_IFNAME等），执行torch.distributed.init_process_group('gloo')时，rank0机器30秒后抛出"Gloo connectFullMesh failed"错误，rank1机器持续挂起。

已完成以下验证：

• nc和telnet测试显示两台机器端口连通性正常；
• 同一机器的两个shell中执行相同配置，分布式初始化可正常完成；
• 尝试重置GLOO_SOCKET_IFNAME等变量，未解决问题。

配置及环境信息 #

环境变量配置 #

rank0机器的env_vars.sh

#!/bin/bash

export MASTER_PORT=23456
export MASTER_ADDR=... # rank0机器的内网IP
export WORLD_SIZE=2
export GLOO_SOCKET_IFNAME=enX0
export RANK=0 

rank1机器的env_vars.sh

#!/bin/bash
export MASTER_PORT=23456
export MASTER_ADDR=... # 同rank0的内网IP
export WORLD_SIZE=2
export GLOO_SOCKET_IFNAME=enX0
export RANK=1

rank0机器网卡信息 #

$ ifconfig
enX0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet ...  netmask 255.255.240.0  broadcast ...
        inet6 ...  prefixlen 64  scopeid 0x20<link>
        ether ...  txqueuelen 1000  (Ethernet)
        RX packets 543929  bytes 577263126 (550.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 203942  bytes 21681067 (20.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 12  bytes 1020 (1020.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 12  bytes 1020 (1020.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

执行步骤 #

$ conda activate pytorch_env
$ . env_vars.sh
$ python
>>> import torch.distributed
>>> torch.distributed.init_process_group('gloo')

错误信息 #

[E ProcessGroupGloo.cpp:138] Gloo connectFullMesh failed with [/opt/conda/conda-bld/pytorch_1699449045860/work/third_party/gloo/gloo/transport/tcp/pair.cc:144] no error
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ec2-user/miniconda3/envs/pytorch_env/lib/python3.9/site-packages/torch/distributed/c10d_logger.py", line 74, in wrapper
    func_return = func(*args, **kwargs)
  File "/home/ec2-user/miniconda3/envs/pytorch_env/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 1155, in init_process_group
    default_pg, _ = _new_process_group_helper(
  File "/home/ec2-user/miniconda3/envs/pytorch_env/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 1293, in _new_process_group_helper
    backend_class = ProcessGroupGloo(backend_prefix_store, group_rank, group_size, timeout=timeout)
RuntimeError: Gloo connectFullMesh failed with [/opt/conda/conda-bld/pytorch_1699449045860/work/third_party/gloo/gloo/transport/tcp/pair.cc:144] no error

连通性测试结果 #

# rank0机器执行
nc -lk 23456
# rank1机器执行
telnet … 23456 # 替换为rank0的内网IP
Trying ...
Connected to …
Escape character is '^]'.
ping
# rank0机器收到输入并显示
ping

排查方案 #

• 验证EC2安全组双向端口规则：Gloo除MASTER_PORT外，会动态分配额外端口用于节点间通信。检查安全组是否开放所有TCP端口（或29500-29600端口范围），且允许两台实例的内网IP双向访问。
• 修正GLOO_SOCKET_IFNAME配置：尝试用网卡的内网IP而非网卡名设置变量，例如export GLOO_SOCKET_IFNAME=10.x.x.x（替换为实例实际内网IP），避免网卡名识别异常。
• 升级/降级PyTorch版本：部分旧版本PyTorch的Gloo后端存在跨节点通信bug，尝试升级到2.0+稳定版本，或降级到1.13等已知无此问题的版本。
• 增加初始化超时时间：在init_process_group中显式设置超时参数：

  import datetime
  torch.distributed.init_process_group('gloo', timeout=datetime.timedelta(seconds=60))
  

• 检查内网IP解析：确认MASTER_ADDR的内网IP在两台实例上均可正常ping通，若存在解析问题，直接使用IP而非主机名。
• 排查实例内部防火墙：执行sudo iptables -L查看规则，必要时临时关闭iptables测试：sudo systemctl stop iptables。
• tcpdump抓包分析：在两台实例同时执行抓包命令：

  tcpdump -i enX0 port 23456 or port range 29500-29600
  

  启动初始化后查看数据包流转情况，定位通信失败环节。

内容的提问来源于stack exchange，提问作者Eric Auld