使用capture_tpu_profile监控TPU任务时连接失败的解决求助

问题 #

我尝试按照Cloud TPU Tools监控任务文档运行以下命令：

capture_tpu_profile --tpu=[my-tpu-name] --monitoring_level=2 --tpu_zone=[my-tpu-zone]

执行后出现错误日志如下：

2022-08-07 08:42:22.253271: I tensorflow/core/tpu/tpu_initializer_helper.cc:66] libtpu.so already in used by another process. Not attempting to load libtpu.so in this process.
WARNING: Logging before InitGoogle() is written to STDERR
I0807 08:42:22.273403  426417 tpu_initializer_helper.cc:66] libtpu.so already in used by another process. Not attempting to load libtpu.so in this process.
2022-08-07 08:42:22.287995: I tensorflow/core/tpu/tpu_initializer_helper.cc:66] libtpu.so already in used by another process. Not attempting to load libtpu.so in this process.
TensorFlow version 2.6.0 detected
Welcome to the Cloud TPU Profiler v2.4.0
I0807 08:42:23.432723 140287849856064 discovery.py:280] URL being requested: GET https://www.googleapis.com/discovery/v1/apis/tpu/v1/rest
I0807 08:42:23.574428 140287849856064 discovery.py:911] URL being requested: GET https://tpu.googleapis.com/v1/projects/[admin name]/locations/europe-west4-a/nodes/[my tpu name]?alt=json
I0807 08:42:23.574645 140287849856064 transport.py:157] Attempting refresh to obtain initial access_token
I0807 08:42:23.626410 140287849856064 discovery.py:280] URL being requested: GET https://www.googleapis.com/discovery/v1/apis/tpu/v1/rest
I0807 08:42:24.250050 140287849856064 discovery.py:911] URL being requested: GET https://tpu.googleapis.com/v1/projects/[admin name]/locations/europe-west4-a/nodes/[my tpu name]?alt=json
I0807 08:42:24.250258 140287849856064 transport.py:157] Attempting refresh to obtain initial access_token
Since monitoring level is provided, profile 10.164.0.16:8466  for  0  ms and show metrics for  100  time(s).
Traceback (most recent call last):
  File "/home/[my name]/.local/bin/capture_tpu_profile", line 8, in <module>
    sys.exit(run_main())
  File "/home/[my name]/.local/lib/python3.8/site-packages/cloud_tpu_profiler/capture_tpu_profile.py", line 136, in run_main
    app.run(main)
  File "/usr/local/lib/python3.8/dist-packages/absl/app.py", line 303, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.8/dist-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "/home/[my name]/.local/lib/python3.8/site-packages/cloud_tpu_profiler/capture_tpu_profile.py", line 184, in main
    monitoring_helper(service_addr, duration_ms, FLAGS.monitoring_level,
  File "/home/[my name]/.local/lib/python3.8/site-packages/cloud_tpu_profiler/capture_tpu_profile.py", line 131, in monitoring_helper
    res = profiler_client.monitor(service_addr, duration_ms, monitoring_level)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/profiler/profiler_client.py", line 167, in monitor
    return _pywrap_profiler.monitor(
tensorflow.python.framework.errors_impl.UnavailableError: failed to connect to all addresses

我已查看过类似问题，但错误情况不同，请问该如何解决？

解决方案 #

1. 清理libtpu.so占用进程 #

日志反复提示libtpu.so already in used by another process，先处理进程占用问题：

• 查找占用libtpu.so的进程：

  lsof | grep libtpu.so
  

• 终止对应进程（替换[PID]为查到的进程ID）：

  kill -9 [PID]
  

• 重启TPU相关任务后，重新执行capture_tpu_profile命令。

2. 验证TPU节点状态与网络连通性 #

• 确认TPU节点处于运行状态，通过GCP控制台检查节点状态，若节点停止或异常，先启动或修复节点。
• 测试本地与TPU节点内部IP的连通性：

  ping 10.164.0.16
  

• 检查VPC防火墙规则，确保开放TPU Profiler默认端口8466，允许本地机器访问TPU节点的内部IP。

3. 对齐TensorFlow与TPU Profiler版本 #

当前TensorFlow版本为2.6.0，TPU Profiler版本为v2.4.0，版本不匹配可能引发兼容性问题：

• 升级TPU Profiler至对应版本：

  pip install --upgrade cloud-tpu-profiler==2.6.*
  

• 确保本地TensorFlow版本与TPU节点的TF版本一致，可通过GCP控制台查看节点的TensorFlow版本，不一致则调整本地版本。

4. 检查账号权限与认证 #

日志显示多次尝试刷新access_token，需确认账号权限：

• 验证当前账号拥有roles/tpu.admin或roles/tpu.viewer权限：

  gcloud projects get-iam-policy [你的项目ID] --filter="bindings.members:[你的邮箱]"
  

• 重新进行账号认证：

  gcloud auth application-default login
  

5. 显式指定项目ID #

在命令中添加--project参数，避免自动识别项目出错：

capture_tpu_profile --tpu=[my-tpu-name] --monitoring_level=2 --tpu_zone=[my-tpu-zone] --project=[你的项目ID]

内容的提问来源于stack exchange，提问作者dinhanhx