运行TensorFlow NLP训练脚本时TPU-VM写入Cloud Bucket遇403权限错误

问题详情 #

运行TensorFlow官方NLP训练脚本train.py时，尝试写入Google Cloud Storage(GCS)存储桶时触发403权限拒绝错误。

执行命令 #

python3 official/nlp/train.py --tpu=con-bert1 --experiment=bert/pretraining --mode=train --model_dir=gs://con_bioberturk/general/ --config_file=gs://con_bioberturk/bert_base.yaml --config_file=gs://con_bioberturk/pretrain.yaml  --params_override="task.init_checkpoint=gs://con_bioberturk/bert-base-turkish-cased-tf/model.ckpt"

报错输出 #

I1115 07:49:02.847452 139877506112576 train_utils.py:368] Saving experiment configuration to gs://con_bioberturk/general/params.yaml
Traceback (most recent call last):
  File "/usr/share/tpu/models/official/modeling/hyperparams/params_dict.py", line 349, in save_params_dict_to_yaml
    yaml.dump(params.as_dict(), f, default_flow_style=False) 

File "/usr/local/lib/python3.8/dist-packages/yaml/__init__.py", line 290, in dump
    return dump_all([data], stream, Dumper=Dumper, **kwds)

File "/usr/local/lib/python3.8/dist-packages/yaml/__init__.py", line 278, in dump_all
    dumper.represent(data)

 File "/usr/local/lib/python3.8/dist-packages/yaml/representer.py", line 28, in represent
    self.serialize(node)
  File "/usr/local/lib/python3.8/dist-packages/yaml/serializer.py", line 55, in serialize
        self.emit(DocumentEndEvent(explicit=self.use_explicit_end))
      File "/usr/local/lib/python3.8/dist-packages/yaml/emitter.py", line 115, in emit
        self.state()
      File "/usr/local/lib/python3.8/dist-packages/yaml/emitter.py", line 220, in expect_document_end
    self.flush_stream()
  File "/usr/local/lib/python3.8/dist-packages/yaml/emitter.py", line 790, in flush_stream
    self.stream.flush()
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/lib/io/file_io.py", line 219, in flush
    self._writable_file.flush()

tensorflow.python.framework.errors_impl.PermissionDeniedError: Error executing an HTTP request: HTTP response code 403 with body '{
  "error": {
    "code": 403,
    "message": "Access denied.",
    "errors": [
      {
        "message": "Access denied.",
        "domain": "global",
        "reason": "forbidden"
      }
    ]
  }
}

    when initiating an upload to gs://con_bioberturk/general/params.yaml

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "official/nlp/train.py", line 82, in <module>
    app.run(main)
  File "/usr/local/lib/python3.8/dist-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.8/dist-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "official/nlp/train.py", line 47, in main
    train_utils.serialize_config(params, model_dir)
  File "/usr/share/tpu/models/official/core/train_utils.py", line 370, in serialize_config
    hyperparams.save_params_dict_to_yaml(params, params_save_path)
  File "/usr/share/tpu/models/official/modeling/hyperparams/params_dict.py", line 349, in save_params_dict_to_yaml
    yaml.dump(params.as_dict(), f, default_flow_style=False)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/lib/io/file_io.py", line 197, in __exit__
    self.close()
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/lib/io/file_io.py", line 239, in close
    self._writable_file.close()
tensorflow.python.framework.errors_impl.PermissionDeniedError: Error executing an HTTP request: HTTP response code 403 with body '{
  "error": {
    "code": 403,
    "message": "Access denied.",
    "errors": [
      {
        "message": "Access denied.",
        "domain": "global",
        "reason": "forbidden"
      }
    ]
  }
}
'            

环境配置 #

• TPU-VM名称：con-bert1
• TPU软件版本：tpu-vm-tf-2.10.0-pod
• Cloud Bucket（con_bioberturk）与TPU-VM位于同一区域

解决步骤 #

1. 确认TPU-VM服务账号权限

   • 在TPU-VM上运行gcloud auth list查看当前活跃服务账号。
   • 给该账号添加Storage Object Creator角色（遵循最小权限原则），确保拥有storage.objects.create和storage.objects.write权限。

2. 检查存储桶权限配置

   • 进入GCS控制台，确认con_bioberturk存储桶的权限列表中，TPU-VM的服务账号被明确授予写入权限，避免依赖自动权限配置。

3. 重新完成VM凭据认证

   • 在TPU-VM上执行gcloud auth application-default login完成本地凭据认证；若使用服务账号密钥，确保GOOGLE_APPLICATION_CREDENTIALS环境变量正确指向密钥文件路径。

4. 测试存储桶写入能力

   • 运行以下命令验证基础写入权限：

     echo "test" > test.txt && gsutil cp test.txt gs://con_bioberturk/general/test.txt
     

   • 若此命令报错，说明核心权限问题未解决，优先排查IAM角色配置。

内容的提问来源于stack exchange，提问作者hazal