无法连接远程Ray.io集群的故障及可行解决办法咨询

问题场景 #

尝试通过ray.init(address="{node_external_ip}:6379")连接远程Ray集群头节点做RPC负载测试，头节点启动命令：

ray start --head --node-ip-address <node-external-IP>

已确认6379端口开放，客户端显示连接成功，但随即报错，头节点也会崩溃。

客户端报错信息 #

Connecting to existing Ray cluster at address: <node-external-IP>:6379...

global_state_accessor.cc:357: This node has an IP address of <client-internal-IP>, while we can not found the matched Raylet address. This maybe come from when you connect the Ray cluster with a different IP address or connect a container.

Connected to Ray cluster.

随后出现：

Failed to get the system config from raylet because it is dead. Worker will terminate. Status: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details:  .Please see `raylet.out` for more details.

远程集群日志 #

• raylet.out：

The raylet exited immediately because the Ray agent failed. The raylet fate shares with the agent. This can happen because the Ray agent was unexpectedly killed or failed. See `dashboard_agent.log` for the root cause.

• dashboard_agent.log：

ERROR agent.py:473 -- Agent is working abnormally. It will exit immediately.
(...)
grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
    status = StatusCode.UNAVAILABLE
    details = "failed to connect to all addresses"
    debug_error_string = "{\"created\":\"@1661955376.270755430\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3134,\"referenced_errors\":[{\"created\":\"@1661955376.270754305\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":163,\"grpc_status\":14}]}"

已尝试的无效操作 #

• 确认dashboard端口开放，使用--include-dashboard false启动结果一致
• 用--block启动头节点，数秒后崩溃，日志相同
• 客户端与头节点使用相同版本Python（3.9.13/3.10.5）和Ray（1.12.0/1.13.0/2.0.0）
• 调用ray.init()时指定_node_ip_address或添加ray://前缀

环境信息 #

• 客户端：Manjaro Linux x86_64，内核5.10.136-1-MANJARO
• 远程集群：Ubuntu 20.04 x86_64（AWS EC2，内核5.13.0-1031-aws）/Manjaro物理机，未使用Docker

可行解决办法 #

• 指定头节点的内外IP映射：启动头节点时同时指定外部IP和内部IP，让Ray能正确识别节点的网络地址，命令示例：

  ray start --head --node-ip-address <external-IP> --ray-node-ip-address <internal-IP>
  

  客户端连接时用外部IP，Ray内部通信走内部IP，避免地址不匹配问题。

• 开放Ray所需的全量端口：除6379外，Ray还依赖6000-6010、8265、10001等端口进行内部通信，确保客户端能访问这些端口。可临时关闭防火墙测试，确认后再配置安全组/防火墙规则。

• 简化Ray启动组件：减少不必要的组件启动，降低连接失败概率，启动命令示例：

  RAY_DISABLE_RESOURCE_MONITOR=1 ray start --head --node-ip-address <external-IP> --include-dashboard false --disable-usage-stats
  

• 使用标准客户端连接协议：客户端连接时用ray://协议指定完整端口，同时显式设置客户端出口IP：

  ray.init(address="ray://<node-external-IP>:10001", _node_ip_address="<client-external-IP>")
  

• 清理Ray残留数据与进程：在头节点执行强制停止命令并删除临时数据，避免旧配置干扰：

  ray stop --force && rm -rf /tmp/ray
  

  之后重新启动集群。

内容的提问来源于stack exchange，提问作者ntonk