Ray集群模式下Worker节点启动后频繁崩溃问题排查求助

环境与启动方式 #

• Ray版本：2.5.1
• Head节点启动命令：ray start --head --port=6379
• Worker节点启动命令：ray start --address=<head-node-address:port>

节点状态变化 #

启动初期（健康状态） #

执行ray status可看到Head节点与Worker节点均处于健康状态：

Node status
---------------------------------------------------------------
Healthy:
 1 node_23703ccdd0ed4574dba6c2806f040b8fac5f881798e658d5e4f27044
 1 node_4693b4303785ad88d5a530088d9ba6db553b4121f36b6106d7e724d6
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/16.0 CPU
 0B/76.88GiB memory
 0B/35.51GiB object_store_memory

Demands:
 (no resource demands)

数秒后（Worker节点崩溃） #

再次执行ray status仅显示Head节点：

======== Autoscaler status: 2023-07-13 18:22:29.514296 ========
Node status
---------------------------------------------------------------
Healthy:
 1 node_4693b4303785ad88d5a530088d9ba6db553b4121f36b6106d7e724d6
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/8.0 CPU
 0B/35.89GiB memory
 0B/17.94GiB object_store_memory

Demands:
 (no resource demands)

Worker节点raylet.err日志内容 #

[2023-07-13 18:21:33,277 C 19488 19488] (raylet) node_manager.cc:1036: [Timeout] Exiting because this node manager has mistakenly been marked as dead by the GCS: GCS failed to check the health of this node for 5 times. This is likely because the machine or raylet has become overloaded.
*** StackTrace Information ***
/data/anaconda/envs/py38_af240/lib/python3.8/site-packages/ray/core/src/ray/raylet/raylet(+0x5361ba) [0x564fec80b1ba] ray::operator<<()
/data/anaconda/envs/py38_af240/lib/python3.8/site-packages/ray/core/src/ray/raylet/raylet(+0x537ba2) [0x564fec80cba2] ray::SpdLogMessage::Flush()
/data/anaconda/envs/py38_af240/lib/python3.8/site-packages/ray/core/src/ray/raylet/raylet(+0x537eb7) [0x564fec80ceb7] ray::RayLog::~RayLog()
/data/anaconda/envs/py38_af240/lib/python3.8/site-packages/ray/core/src/ray/raylet/raylet(+0x29bb13) [0x564fec570b13] ray::raylet::NodeManager::NodeRemoved()
/data/anaconda/envs/py38_af240/lib/python3.8/site-packages/ray/core/src/ray/raylet/raylet(+0x40b0a7) [0x564fec6e00a7] ray::gcs::NodeInfoAccessor::HandleNotification()
/data/anaconda/envs/py38_af240/lib/python3.8/site-packages/ray/core/src/ray/raylet/raylet(+0x515166) [0x564fec7ea166] EventTracker::RecordExecution()
/data/anaconda/envs/py38_af240/lib/python3.8/site-packages/ray/core/src/ray/raylet/raylet(+0x4a262e) [0x564fec77762e] std::_Function_handler<>::_M_invoke()
/data/anaconda/envs/py38_af240/lib/python3.8/site-packages/ray/core/src/ray/raylet/raylet(+0x4a2b86) [0x564fec777b86] boost::asio::detail::completion_handler<>::do_complete()
/data/anaconda/envs/py38_af240/lib/python3.8/site-packages/ray/core/src/ray/raylet/raylet(+0xa97cfb) [0x564fecd6ccfb] boost::asio::detail::scheduler::do_run_one()
/data/anaconda/envs/py38_af240/lib/python3.8/site-packages/ray/core/src/ray/raylet/raylet(+0xa9a289) [0x564fecd6f289] boost::asio::detail::scheduler::run()
/data/anaconda/envs/py38_af240/lib/python3.8/site-packages/ray/core/src/ray/raylet/raylet(+0xa9a742) [0x564fecd6f742] boost::asio::io_context::run()
/data/anaconda/envs/py38_af240/lib/python3.8/site-packages/ray/core/src/ray/raylet/raylet(+0x168ff2) [0x564fec43dff2] main
/lib64/libc.so.6(__libc_start_main+0xea) [0x7fdc1d9ed13a] __libc_start_main
/data/anaconda/envs/py38_af240/lib/python3.8/site-packages/ray/core/src/ray/raylet/raylet(+0x1b0567) [0x564fec485567]

排查现状与求助 #

已确认Head节点与Worker节点均无其他重载进程，现寻求排查该问题的建议。

内容的提问来源于stack exchange，提问作者user3435903