Deepspeed两节点分布式训练遇NCCL连接超时错误求助

云上老周

2026-7-19

两节点分布式训练NCCL连接超时错误排查请求

使用Deepspeed、PyTorch和Transformers进行两节点分布式训练，数据预处理完成、即将启动训练时触发NCCL错误，具体错误日志如下：

[0] NCCL INFO cudaDriverVersion 12000
[0] NCCL INFO Bootstrap : Using ens5:172.x.x.x<0>
[0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
[0] NCCL INFO init.cc:1147 Cuda Host Alloc Size 4 pointer 0x7f23fca00000
[0] NCCL INFO NET/IB : No device found.
[0] NCCL INFO NET/Socket : Using [0]ens5:172.x.x.x<0>
[0] NCCL INFO Using network Socket
[0] misc/socket.cc:456 NCCL WARN Net : Connect to 172.x.x.x<47961> failed : Connection timed out
[0] NCCL INFO bootstrap.cc:256 -> 6
[0] NCCL INFO init.cc:516 -> 6
[0] NCCL INFO init.cc:1089 -> 6
[0] NCCL INFO group.cc:64 -> 6 [Async thread]
[0] NCCL INFO group.cc:421 -> 3
[0] NCCL INFO group.cc:106 -> 3
[0] NCCL INFO comm 0x55d4887754d0 rank 1 nranks 2 cudaDev 0 busId 1e0 - Abort COMPLETE
Traceback (most recent call last):
:   File "/home/ubuntu/run_clm.py", line 635, in <module>
:     main()
:   File "/home/ubuntu/run_clm.py", line 455, in main
:     with training_args.main_process_first(desc="dataset map tokenization"):
:   File "/opt/conda/envs/test_env/lib/python3.11/contextlib.py", line 144, in __exit__
:     next(self.gen)
:   File "/opt/conda/envs/test_env/lib/python3.11/site-packages/transformers/training_args.py", line 1902, in main_process_first
:     torch.distributed.barrier()
:   File "/opt/conda/envs/test_env/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 3328, in barrier
:     work = default_pg.barrier(opts=opts)
:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1275, internal error, NCCL version 2.14.3
: ncclInternalError: Internal check failed.
: Last error:
: Net : Connect to 172.x.x.x<47961> failed : Connection timed out
: [2023-06-04 09:59:10,288] [INFO] [launch.py:428:sigkill_handler] Killing subprocess 17994

环境配置：