Google Cloud TPU VM /tmp/tpu_logs日志权限拒绝问题排查

参照mesh-transformer-jax仓库的指引微调GPT-J模型，此前曾在同一台Google Cloud TPU VM上完成过一次模型微调，后续删除了对应的微调模型文件。现在在该VM上启动新的微调任务时，代码尝试访问tpu_logs相关文件抛出权限不足报错，确认是否有其他用户遇到过同类问题。

终端反复弹出大量同类报错：
Could not open the log file '/tmp/tpu_logs/tpu_driver.[vm-code].[my-account].log.INFO.20220623-164754.59430': Permission denied

最初怀疑是代码库识别到历史微调痕迹，尝试自动断点续训导致异常，现寻求该问题的有效解决方案。

完整运行输出如下：

.
.
.
Could not open the log file '/tmp/tpu_logs/tpu_driver.[vm-code].[my-account].log.INFO.20220623-164754.59430': Permission denied
Could not open any log file.
jax devices: 8
jax runtime initialized in 3.53237s
`--tune_model_path` passed: we are beginning a fine-tuning run
path to load checkpoint from: gs://[my-bucket]/step_383500/
setting up datasets
initializing network
/home/[my-username]/miniconda3/envs/finetune-env/lib/python3.8/site-packages/jax/experimental/maps.py:527: UserWarning: xmap is an experimental feature and probably has bugs!
  warn("xmap is an experimental feature and probably has bugs!")
/home/[my-username]/miniconda3/envs/finetune-env/lib/python3.8/site-packages/jax/_src/lib/xla_bridge.py:429: UserWarning: jax.host_count has been renamed to jax.process_count. This alias will eventually be removed; please update your code.
  warnings.warn(
Could not open the log file '/tmp/tpu_logs/tpu_driver.[vm-code].[my-account].log.INFO.20220623-164754.59430': Permission denied
.
.
.
Could not open any log file.
Could not open the log file '/tmp/tpu_logs/tpu_driver.[vm-code].[my-account].log.INFO.20220623-164754.59430': Permission denied
Could not open any log file.
/home/[my-username]/miniconda3/envs/finetune-env/lib/python3.8/site-packages/jax/_src/lib/xla_bridge.py:416: UserWarning: jax.host_id has been renamed to jax.process_index. This alias will eventually be removed; please update your code.
  warnings.warn(
key shape (8, 2)
in shape (1, 2048)
dp 1
mp 8
Could not open the log file '/tmp/tpu_logs/tpu_driver.[vm-code].[my-account].log.INFO.20220623-164754.59430': Permission denied
Could not open any log file.
.
.
.
Could not open any log file.
Could not open the log file '/tmp/tpu_logs/tpu_driver.[vm-code].[my-account].log.INFO.20220623-164754.59430': Permission denied
Could not open any log file.
Total parameters: 6053381344
loading network
Could not open the log file '/tmp/tpu_logs/tpu_driver.[vm-code].[my-account].log.INFO.20220623-164754.59430': Permission denied
Could not open any log file.
Could not open the log file '/tmp/tpu_logs/tpu_driver.[vm-code].[my-account].log.INFO.20220623-164754.59430': Permission denied
.
.
.

补充排查结果：经确认，该日志权限报错在执行import tensorflow as tf时就会触发。

这个问题和断点续训没有关系，核心原因是/tmp/tpu_logs目录权限配置错误：之前运行任务时该目录大概率被root权限的进程创建或写入，当前运行环境的普通用户没有目录的读写权限，才会反复抛出日志写入失败的报错。

• 快速修复可以直接执行命令开放目录权限：
  sudo chmod -R 777 /tmp/tpu_logs
  如果不需要开放全局权限，也可以直接将目录所有者修改为当前登录用户：
  sudo chown -R $USER:$USER /tmp/tpu_logs
• 如果修改权限后仍然报错，直接删除整个目录即可，TPU驱动下次启动时会自动创建符合权限要求的新日志目录：
  sudo rm -rf /tmp/tpu_logs
• 补充说明：这个报错仅代表日志写入失败，不会影响TPU的实际计算逻辑，也不会阻碍微调任务正常运行，只是反复刷报错信息影响终端观感，修改完目录权限后报错就会消失。运行输出中提到的xmap为实验特性、jax.host_count重命名的提示都属于版本兼容类的正常警告，不影响任务运行，不需要额外处理。

内容的提问来源于stack exchange，提问作者Jacques Thibodeau