TensorFlow Cloud ML目标检测分布式训练异常问题求助

我正在遵循TensorFlow的分布式训练教程训练自定义模型，完全使用官方Object Detection仓库中的代码。和教程相比我做了一处主要改动：将runtime版本从1.2升级到了1.5。

在Google Cloud ML上运行时，没有明确的报错信息，但任务很快就退出了，完全没开始训练。

启动训练的命令 #

gcloud ml-engine jobs submit training object_detection_`date +%s` --job-dir=gs://test-bucket/training/ --packages dist/object_detection-0.1.tar.gz,slim/dist/slim-0.1.tar.gz --module-name object_detection.train --region us-central1 --config ./config.yaml -- --train_dir=gs://test-bucket/data/ --pipeline_config_path=gs://test-bucket/configs/ssd_inception_v2_coco.config

config.yaml配置内容 #

trainingInput:
  runtimeVersion: "1.5"
  scaleTier: CUSTOM
  masterType: complex_model_l
  workerCount: 9
  workerType: standard_gpu
  parameterServerCount: 3
  parameterServerType: large_model

任务结束时的日志片段 #

I worker-replica-6 Clean up finished. worker-replica-6
I worker-replica-7 Signal 15 (SIGTERM) was caught. Terminated by service. This is normal behavior. worker-replica-7
I worker-replica-7 Module completed; cleaning up. worker-replica-7
I worker-replica-7 Clean up finished. worker-replica-7
I worker-replica-8 Signal 15 (SIGTERM) was caught. Terminated by service. This is normal behavior. worker-replica-8
I worker-replica-8 Module completed; cleaning up. worker-replica-8
I worker-replica-8 Clean up finished. worker-replica-8
I worker-replica-1 CreateSession still waiting for response from worker: /job:master/replica:0/task:0 worker-replica-1
I worker-replica-1 Signal 15 (SIGTERM) was caught. Terminated by service. This is normal behavior. worker-replica-1
I worker-replica-1 Module completed; cleaning up. worker-replica-1
I worker-replica-1 Clean up finished. worker-replica-1
I worker-replica-7 CreateSession still waiting for response from worker: /job:master/replica:0/task:0 worker-replica-7
I worker-replica-8 CreateSession still waiting for response from worker: /job:master/replica:0/task:0 worker-replica-8
I worker-replica-6 CreateSession still waiting for response from worker: /job:master/replica:0/task:0 worker-replica-6
I worker-replica-3 CreateSession still waiting for response from worker: /job:master/replica:0/task:0 worker-replica-3
I worker-replica-0 CreateSession still waiting for response from worker: /job:master/replica:0/task:0 worker-replica-0
I worker-replica-2 CreateSession still waiting for response from worker: /job:master/replica:0/task:0 worker-replica-2
I worker-replica-5 CreateSession still waiting for response from worker: /job:master/replica:0/task:0 worker-replica-5
I worker-replica-1 CreateSession still waiting for response from worker: /job:master/replica:0/task:0 worker-replica-1
I worker-replica-7 CreateSession still waiting for response from worker: /job:master/replica:0/task:0 worker-replica-7
I worker-replica-8 CreateSession still waiting for response from worker: /job:master/replica:0/task:0 worker-replica-8
I worker-replica-6 CreateSession still waiting for response from worker: /job:master/replica:0/task:0 worker-replica-6
I worker-replica-3 CreateSession still waiting for response from worker: /job:master/replica:0/task:0 worker-replica-3
I worker-replica-0 CreateSession still waiting for response from worker: /job:master/replica:0/task:0 worker-replica-0
I worker-replica-2 CreateSession still waiting for response from worker: /job:master/replica:0/task:0 worker-replica-2
I worker-replica-5 CreateSession still waiting for response from worker: /job:master/replica:0/task:0 worker-replica-5
I worker-replica-1 CreateSession still waiting for response from worker: /job:master/replica:0/task:0 worker-replica-1
I worker-replica-7 CreateSession still waiting for response from worker: /job:master/replica:0/task:0 worker-replica-7
I worker-replica-8 CreateSession still waiting for response from worker: /job:master/replica:0/task:0 worker-replica-8
I worker-replica-6 CreateSession still waiting for response from worker: /job:master/replica:0/task:0 worker-replica-6
I Finished tearing down TensorFlow.
I Job failed.

日志稍早位置还出现了错误：Master init: Unavailable: Stream removed，但我不知道该怎么处理这个问题，希望能得到解决方向！

根据你遇到的问题和日志信息，这里有几个可以排查的方向：

1. 检查Master实例的资源配置 #

你使用的complex_model_l实例在TensorFlow 1.5环境下，可能无法支撑初始化阶段的资源需求，导致Master节点提前崩溃，进而让所有Worker节点因为无法连接Master而被终止。

可以尝试：

• 先降低集群规模，比如把workerCount改成1，parameterServerCount改成1，用最小集群验证是否能正常启动训练
• 如果最小集群能运行，再逐步增加Worker数量；如果还是不行，尝试升级Master实例类型到更高级别（比如standard_p100）

2. 验证TensorFlow版本兼容性 #

TensorFlow 1.5相对于1.2有不少底层变化，Object Detection仓库的代码可能需要对应版本的适配：

• 确认你使用的Object Detection仓库代码是对应TensorFlow 1.5的分支版本（比如找tags为r1.5的代码）
• 检查ssd_inception_v2_coco.config配置文件中的参数是否和TF1.5兼容，比如某些分布式训练相关的配置项是否需要调整

3. 排查GCS存储权限和路径问题 #

日志中没有明确的权限报错，但Master节点初始化失败可能和GCS路径访问有关：

• 确认提交任务的服务账号（默认是ML Engine的服务账号）拥有gs://test-bucket的读写权限
• 检查train_dir和pipeline_config_path的路径是否正确，有没有拼写错误，文件是否存在

4. 查看更详细的Master节点日志 #

目前你只提供了Worker节点的日志，Master节点的崩溃日志才是关键：

• 登录Google Cloud Console的ML Engine页面，找到对应的任务，查看Master节点的完整日志，应该能找到Master init: Unavailable: Stream removed的上下文信息，这会帮你定位具体的初始化失败原因

内容的提问来源于stack exchange，提问作者kevin.w.johnson