Colab Pro+训练DL模型时出现Transport endpoint is not connected错误求助

问题背景 #

我是Google Colab Pro+用户，训练深度学习模型时总会触发「Transport endpoint is not connected」错误。使用Pro版本时就出现过该问题，升级到Pro+后依然未解决。代码运行3-4小时左右必然出错，已尝试用JavaScript维持运行时在线，但问题依旧。

完整错误信息：

Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
 18% 54466/300000 [3:57:19<17:49:51,  3.83it/s]
Traceback (most recent call last):
  File "/content/drive/MyDrive/Ub4D/exp_runner.py", line 998, in <module>
  File "/content/drive/MyDrive/Ub4D/exp_runner.py", line 257, in train
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/tensorboard/writer.py", line 391, in add_scalar
    self._get_file_writer().add_summary(summary, global_step, walltime)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/tensorboard/writer.py", line 113, in add_summary
    self.run()
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/summary/writer/event_file_writer.py", line 244, in run
    self.add_event(event, global_step, walltime)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/tensorboard/writer.py", line 98, in add_event
    self._run()
    self.event_writer.add_event(event)
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/summary/writer/event_file_writer.py", line 117, in add_event
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/summary/writer/event_file_writer.py", line 275, in _run
    self._record_writer.write(data)
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/summary/writer/record_writer.py", line 40, in write
    self._async_writer.write(event.SerializeToString())
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/summary/writer/event_file_writer.py", line 171, in write
    self._writer.write(header + header_crc + data + footer_crc)
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/lib/io/file_io.py", line 101, in write
    self._check_worker_status()
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/summary/writer/event_file_writer.py", line 212, in _check_worker_status
    self._writable_file.append(
    raise exception
tensorflow.python.framework.errors_impl.FailedPreconditionError: exp/Cactus/paper_config/logs/events.out.tfevents.1691166619.f28574666dfe.12707.0; Transport endpoint is not connected
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/summary/writer/event_file_writer.py", line 244, in run
    self._run()
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/summary/writer/event_file_writer.py", line 275, in _run
    self._record_writer.write(data)
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/summary/writer/record_writer.py", line 40, in write
    self._writer.write(header + header_crc + data + footer_crc)
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/lib/io/file_io.py", line 101, in write
    self._writable_file.append(
tensorflow.python.framework.errors_impl.FailedPreconditionError: exp/Cactus/paper_config/logs/events.out.tfevents.1691166619.f28574666dfe.12707.0; Transport endpoint is not connected

解决方案 #

该错误本质是Colab运行时与Google Drive的挂载连接超时中断，导致无法写入TensorBoard日志等文件。以下是有效解决方法：

• 将数据与输出移至Colab本地磁盘
  避免直接读写挂载的Drive文件，先把数据集、代码复制到/content目录，训练日志、模型checkpoint先存本地，结束后再同步回Drive。示例命令：

  # 复制代码到本地
  cp -r /content/drive/MyDrive/Ub4D /content/
  # 训练完成后同步结果回Drive
  cp -r /content/Ub4D/exp /content/drive/MyDrive/Ub4D/
  

• 捕获连接异常并重新挂载Drive
  在代码中用try-except块包裹文件写入操作，捕获FailedPreconditionError时重新挂载Drive并初始化相关写入对象：

  from google.colab import drive
  import tensorflow as tf
  from torch.utils.tensorboard import SummaryWriter
  
  def safe_add_scalar(writer, tag, scalar_value, global_step):
      try:
          writer.add_scalar(tag, scalar_value, global_step)
      except tf.errors.FailedPreconditionError:
          # 重新挂载Drive
          drive.mount('/content/drive', force_remount=True)
          # 重新初始化TensorBoard writer
          writer = SummaryWriter('/content/drive/MyDrive/Ub4D/exp/Cactus/paper_config/logs')
          writer.add_scalar(tag, scalar_value, global_step)
  

• 禁用TensorBoard异步写入
  异步写入会加剧Drive连接不稳定问题，改用同步写入模式，初始化writer时设置flush_secs=1：

  from torch.utils.tensorboard import SummaryWriter
  
  # 每1秒同步刷新一次日志
  writer = SummaryWriter('/content/drive/MyDrive/Ub4D/exp/Cactus/paper_config/logs', flush_secs=1)
  

• 降低Drive读写频率
  用Colab本地临时存储（/tmp或/content）保存训练中间文件，仅在关键节点（如每10个epoch）将checkpoint同步到Drive，减少挂载连接的压力。

内容的提问来源于stack exchange，提问作者Albert