使用Docker搭配Ray时出现protocol version mismatch错误求助

错误日志 #

Digest: sha256:2c0f86c6209e02d1f9e3d0a611ac04d73c3ed132b8b9319a28e2069f1503fb53
Status: Downloaded newer image for us-west1-docker.pkg.dev/project-id/sleep-detection/v1:latest
us-west1-docker.pkg.dev/project-id/sleep-detection/v1:latest
Shared connection to 34.82.93.47 closed.
Shared connection to 34.82.93.47 closed.
Shared connection to 34.82.93.47 closed.
Shared connection to 34.82.93.47 closed.
Shared connection to 34.82.93.47 closed.
WARNING: The requested image's platform (linux/amd64/v2) does not match the detected host platform (linux/amd64) and no specific platform was requested
b2567e6280d5124da59fec72d17f89e15ceaa866b991830c8a747df988334e60
Shared connection to 34.82.93.47 closed.
Shared connection to 34.82.93.47 closed.
protocol version mismatch -- is your shell clean?
(see the rsync man page for an explanation)
rsync error: protocol incompatibility (code 2) at compat.c(178) [sender=3.1.3]
Shared connection to 34.82.93.47 closed.
2023-11-26 20:17:52,373 INFO node.py:321 -- wait_for_compute_zone_operation: Waiting for operation operation-1701051472184-60b18e8be856a-ab18acec-f1df8579 to finish...
2023-11-26 20:17:57,644 INFO node.py:340 -- wait_for_compute_zone_operation: Operation operation-1701051472184-60b18e8be856a-ab18acec-f1df8579 finished.
  New status: update-failed
  !!!
  SSH command failed.
  !!!

  Failed to setup head node.

集群配置 #

# A unique identifier for the head node and workers of this cluster.
cluster_name: minimal
max_workers: 1
# Cloud-provider specific configuration.
provider:
    type: gcp
    region: us-west1
    project_id: [project-id]
    availability_zone: us-west1-a

docker:
  image: us-west1-docker.pkg.dev/[project-id]/sleep-detection/v1
  container_name: test
  pull_before_run: true
  run_options:
    - --detach

available_node_types:
  ray_head_default:
    node_config:
      machineType: e2-standard-2
      disks:
        - boot: true
          autoDelete: true
          initializeParams:
            diskSizeGb: 50
            sourceImage: projects/ml-images/global/images/c0-deeplearning-common-cpu-v20230925-debian-10

      serviceAccounts:
        - email: ray-autoscaler-sa-v1@[project-id].iam.gserviceaccount.com
          scopes:
            - https://www.googleapis.com/auth/cloud-platform

    resources:
      CPU: 2

  ray_worker_default:
    min_workers: 0
    max_workers: 1
    node_config:
      machineType: e2-standard-2
      disks:
        - boot: true
          autoDelete: true
          initializeParams:
            diskSizeGb: 10
            sourceImage: projects/ubuntu-os-cloud/global/images/ubuntu-2304-lunar-amd64-v20231030
            diskType: projects/ubuntu-os-cloud/zones/us-west1-a/diskTypes/pd-standard
      serviceAccounts:
        - email: ray-autoscaler-sa-v1@[project-id].iam.gserviceaccount.com
          scopes:
            - https://www.googleapis.com/auth/cloud-platform
    resources:
      CPU: 2

head_node_type: ray_head_default

initialization_commands:
  - gcloud auth configure-docker us-west1-docker.pkg.dev --quiet

排查与解决步骤 #

1. 修复rsync协议不兼容问题 #

日志中protocol version mismatch -- is your shell clean?是核心故障，原因是SSH连接时shell输出了额外内容（如欢迎信息、环境变量提示），干扰了rsync通信。

• 检查头节点的~/.bashrc、~/.profile等配置文件，注释所有产生输出的语句（比如echo、带提示的export）。
• 在初始化命令中添加清理shell输出的指令：

  - sed -i '/^echo/d' ~/.bashrc
  - sed -i '/^export.*prompt/d' ~/.bashrc
  

2. 解决Docker平台兼容性警告 #

WARNING: The requested image's platform (linux/amd64/v2) does not match the detected host platform (linux/amd64)可能导致容器运行异常，间接影响SSH流程：

• 在Docker配置块中指定平台参数：

  docker:
    # ... 其他配置
    run_options:
      - --detach
      - --platform linux/amd64
  

• 或者重新构建镜像时指定平台：

  docker build --platform linux/amd64 -t us-west1-docker.pkg.dev/[project-id]/sleep-detection/v1 .
  

3. 统一节点基础镜像 #

当前头节点用Debian 10 ML镜像，工作节点用Ubuntu 23.04，系统环境差异可能引发依赖不兼容：

• 将头节点和工作节点的sourceImage统一为同一系统版本（比如都用Ubuntu 23.04或Debian 10）。

4. 确保Docker认证在容器内生效 #

当前gcloud auth configure-docker是在主机层面执行，容器内可能无法拉取私有镜像：

• 将认证命令移到docker_setup_commands中，确保容器内完成认证：

  docker:
    # ... 其他配置
    docker_setup_commands:
      - gcloud auth configure-docker us-west1-docker.pkg.dev --quiet
  

内容的提问来源于stack exchange，提问作者Prabhjot Singh Rai