Ray集群无法通过URL连接Head Node GCS问题求助

环境与部署情况 #

• 托管K8s环境部署Ray集群：1个Head Node + 2个Worker Node
• 通过Service和Ingress暴露端口：
  • 8265（Dashboard）→ http://head-node-dashboard.company.internal.domain.com
  • 6379（GCS）→ http://head-node-gcs.company.internal.domain.com
  • Ray Client（10001）、Serve（8000）也配置了对应内部域名

正常运行场景 #

使用Dashboard域名提交任务可正常执行：

ray job submit --working-dir ./ --address='http://head-node-dashboard.company.internal.domain.com' -- python ./script.py

GCS连接失败场景 #

1. Worker Node通过ray start连接Head Node #

$ > ray start --address='head-node-gcs.company.internal.domain.com:80'
Local node IP: 10.251.222.101
2023-03-18 06:51:17,521 WARNING utils.py:1446 -- Unable to connect to GCS at head-node-gcs.company.internal.domain.com:80. Check that (1) Ray GCS with matching version started successfully at the specified address, and (2) there is no firewall setting preventing access.

2. 通过ray.init()连接Head Node #

a) 不指定协议

>>> ray.init(address='head-node-gcs.company.internal.domain.com:80')
2023-03-18 06:58:11,670 INFO worker.py:1333 -- Connecting to existing Ray cluster at address: head-node-gcs.company.internal.domain.com:80...
2023-03-18 06:58:16,743 WARNING utils.py:1333 -- Unable to connect to GCS at head-node-gcs.company.internal.domain.com:80. Check that (1) Ray GCS with matching version started successfully at the specified address, and (2) there is no firewall setting preventing access.

b) 指定http://协议

>>> ray.init(address='http://head-node-gcs.company.internal.domain.com')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/project/workSpace/venv/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/project/workSpace/venv/lib/python3.8/site-packages/ray/_private/worker.py", line 1230, in init
    builder = ray.client(address, _deprecation_warn_enabled=False)
  File "/project/workSpace/venv/lib/python3.8/site-packages/ray/client_builder.py", line 382, in client
    builder = _get_builder_from_address(address)
  File "/project/workSpace/venv/lib/python3.8/site-packages/ray/client_builder.py", line 350, in _get_builder_from_address
    assert "ClientBuilder" in dir(
AssertionError: Module: http does not have ClientBuilder.

c) 指定ray://协议

>>> ray.init(address='ray://head-node-gcs.company.internal.domain.com')
/project/workSpace/venv/lib/python3.8/site-packages/ray/util/client/worker.py:253: UserWarning: Ray Client connection timed out. Ensure that the Ray Client port on the head node is reachable from your local machine. See https://docs.ray.io/en/latest/cluster/ray-client.html#step-2-check-ports for more information.
  warnings.warn(
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/project/workSpace/venv/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/project/workSpace/venv/lib/python3.8/site-packages/ray/_private/worker.py", line 1248, in init
    ctx = builder.connect()
  File "/project/workSpace/venv/lib/python3.8/site-packages/ray/client_builder.py", line 178, in connect
    client_info_dict = ray.util.client_connect.connect(
  File "/project/workSpace/venv/lib/python3.8/site-packages/ray/util/client_connect.py", line 47, in connect
    conn = ray.connect(
  File "/project/workSpace/venv/lib/python3.8/site-packages/ray/util/client/__init__.py", line 252, in connect
    conn = self.get_context().connect(*args, **kw_args)
  File "/project/workSpace/venv/lib/python3.8/site-packages/ray/util/client/__init__.py", line 94, in connect
    self.client_worker = Worker(
  File "/project/workSpace/venv/lib/python3.8/site-packages/ray/util/client/worker.py", line 139, in __init__
    self._connect_channel()
  File "/project/workSpace/venv/lib/python3.8/site-packages/ray/util/client/worker.py", line 260, in _connect_channel
    raise ConnectionError("ray client connection timeout")
ConnectionError: ray client connection timeout

正常连接场景（使用Head Node本地IP） #

$ > ray init --address='10.251.222.100:6379'
Local node IP: 10.251.222.101
2023-03-18 07:20:41,943 WARNING services.py:1791 -- WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 67108864 bytes available. This will harm performance! You may be able to free up space by deleting files in /dev/shm. If you are inside a Docker container, you can increase /dev/shm size by passing '--shm-size=2.47gb' to 'docker run' (or add it to the run_options list in a Ray cluster config). Make sure to set this to more than 30% of available RAM.
[2023-03-18 07:20:41,964 I 115596 115596] global_state_accessor.cc:356: This node has an IP address of 10.251.222.101, while we can not find the matched Raylet address. This maybe come from when you connect the Ray cluster with a different IP address or connect a container.

--------------------
Ray runtime started.
--------------------

To terminate the Ray runtime, run
  ray stop

期望结果 #

实现ray init --address='head-node-gcs.company.internal.domain.com:80'正常连接Ray集群。

相关配置 #

Head Node Service配置 #

ports:
  - name: ray-dashboard
    port: 8265
    targetPort: 8265
    protocol: TCP
  - name: ray-gcs
    port: 6379
    targetPort: 6379
    protocol: TCP
  - name: ray-client
    port: 10001
    targetPort: 10001
    protocol: TCP
  - name: ray-serve
    port: 8000
    targetPort: 8000
    protocol: TCP
  type: ClusterIP

Ingress配置 #

spec:
  rules:
  - host: head-node-dashboard.company.internal.domain.com
    http:
      paths:
      - path: /
        backend:
          serviceName: head-node-svc
          servicePort: 8265
  - host: head-node-gcs.company.internal.domain.com
    http:
      paths:
      - path: /
        backend:
          serviceName: head-node-svc
          servicePort: 6379
  - host: head-node-client.company.internal.domain.com
    http:
      paths:
      - path: /
        backend:
          serviceName: head-node-svc
          servicePort: 10001
  - host: head-node-serve.company.internal.domain.com
    http:
      paths:
      - path: /
        backend:
          serviceName: head-node-svc
          servicePort: 8000

环境版本 #

$ > ray --version
ray, version 2.3.0

$ > python --version
Python 3.7.4

$ > uname -a
Linux head-node-659568794c-rwmpk 3.10.0-1160.el7.x86_64 #1 SMP Tue Aug 18 14:50:17 EDT 2020 x86_64 GNU/Linux

问题分析与解决方案 #

核心原因 #

Ray的GCS服务使用原生TCP协议通信，当前配置的Ingress是基于HTTP协议的转发，无法直接透传TCP流量到GCS端口。Dashboard能正常工作是因为它本身就是HTTP服务，适配Ingress的HTTP转发规则。

解决方案 #

方案1：集群内部直接使用Service名称访问

如果Worker Node和Head Node在同一K8s集群内，无需通过Ingress，直接用Service名称加端口访问GCS：

ray start --address='head-node-svc:6379'

代码中连接：

ray.init(address='head-node-svc:6379')

K8s集群内部会自动解析Service名称到ClusterIP，无需额外配置。

方案2：配置TCP类型Ingress（跨集群/外部访问）

若必须通过域名访问GCS，需修改Ingress配置添加TCP端口转发（以NGINX Ingress为例）：

1. 创建TCP转发ConfigMap：

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-ingress-tcp
data:
  80: "default/head-node-svc:6379" # 格式：外部端口:命名空间/服务名:目标端口

2. 更新Ingress Controller Deployment，挂载该ConfigMap并启用TCP转发：
   在Deployment的spec.template.spec.containers中添加：

ports:
- name: tcp-80
  containerPort: 80
  protocol: TCP
volumeMounts:
- name: tcp-config
  mountPath: /etc/nginx/tcp-services.d/
volumes:
- name: tcp-config
  configMap:
    name: nginx-ingress-tcp

3. 确认域名head-node-gcs.company.internal.domain.com解析到Ingress Controller的外部IP，之后即可使用目标命令连接：

ray init --address='head-node-gcs.company.internal.domain.com:80'

方案3：使用NodePort/LoadBalancer类型Service

将Head Node的Service类型改为NodePort：

type: NodePort
ports:
- name: ray-gcs
  port: 6379
  targetPort: 6379
  nodePort: 30001 # 可指定端口，或让K8s自动分配

之后通过NodeIP+NodePort访问：

ray start --address='node-ip:30001'

云环境中也可使用LoadBalancer类型，通过分配的外部IP+端口访问。

内容的提问来源于stack exchange，提问作者shinvu