微调BERT分类模型时创建Checkpoint遇PermissionError求助

问题描述 #

微调BERT模型用于文本分类任务时，训练流程在创建第一个checkpoint前运行正常，但之后抛出PermissionError，报错无法访问BERT-emotion-classification\\checkpoint-500目录。已确认目标文件夹拥有全部权限，但问题依然存在。

训练代码 #

from transformers import Trainer, TrainingArguments, AutoConfig
from transformers import Trainer

batch_size = 8
logging_steps = len(emotions_encoded['train']) // batch_size
print(len(emotions_encoded['train']))
print(logging_steps)

model_name = f"BERT-emotion-classification"

# Create a configuration object
config = AutoConfig.from_pretrained(model_ckpt, output_hidden_states=True)

# Save the configuration to a JSON file
config.to_json_file(f"{model_name}/config.json")

training_args = TrainingArguments(output_dir=model_name,
                                        num_train_epochs=2,
                                        learning_rate=2e-5,
                                        per_device_train_batch_size=batch_size,
                                        per_device_eval_batch_size=batch_size,
                                        weight_decay=0.01,
                                        evaluation_strategy="epoch",
                                        disable_tqdm=False,
                                        logging_steps=logging_steps,
                                        push_to_hub=True,
                                        log_level="error")


trainer = Trainer(model=model, args=training_args,
                                compute_metrics=compute_metrics,
                                train_dataset=emotions_encoded["train"],
                                eval_dataset=emotions_encoded["validation"],
                                tokenizer=tokenizer)
trainer.train();

# Save the model using Trainer's save_model method
trainer.save_model(f"./{model_name}")

错误日志 #

{
    "name": "PermissionError",
    "message": "[Errno 13] Permission denied: 'BERT-emotion-classification\\\\checkpoint-500'",
    "stack": "---------------------------------------------------------------------------
PermissionError                           Traceback (most recent call last)
Cell In[34], line 7
      1 from transformers import Trainer
      2 trainer = Trainer(model=model, args=training_args,
      3                                 compute_metrics=compute_metrics,
      4                                 train_dataset=emotions_encoded[\"train\"],
      5                                 eval_dataset=emotions_encoded[\"validation\"],
      6                                 tokenizer=tokenizer)
----> 7 trainer.train();
      9 # Save the model using Trainer's save_model method
     10 trainer.save_model(f\"./{model_name}\")

PermissionError: [Errno 13] Permission denied: 'BERT-emotion-classification\\\\checkpoint-500'"
}

解决方案 #

• 手动预创建checkpoint目录
  虽然主目录权限正常，但checkpoint子目录的权限可能未正确继承。手动创建BERT-emotion-classification/checkpoint-500目录后再启动训练，验证是否还会触发权限错误。同时确保路径中无空格、非ASCII特殊字符，Windows系统下优先用正斜杠/或双反斜杠\\作为路径分隔符。

• 释放目录锁定进程
  文件管理器、IDE（如Jupyter Notebook）或后台进程可能锁定了目标目录，导致Trainer无法写入。关闭所有可能访问该目录的程序，或重启训练环境后再尝试。

• 调整Checkpoint保存策略
  修改TrainingArguments中的checkpoint相关参数，降低写入频率或限制保存数量：

  training_args = TrainingArguments(output_dir=model_name,
                                  num_train_epochs=2,
                                  learning_rate=2e-5,
                                  per_device_train_batch_size=batch_size,
                                  per_device_eval_batch_size=batch_size,
                                  weight_decay=0.01,
                                  evaluation_strategy="epoch",
                                  disable_tqdm=False,
                                  logging_steps=logging_steps,
                                  push_to_hub=True,
                                  log_level="error",
                                  save_strategy="epoch",  # 改为按epoch保存checkpoint
                                  save_total_limit=2)  # 限制最多保存2个checkpoint
  

• 移除手动保存config的代码
  代码中提前用config.to_json_file创建了配置文件，可能与Trainer保存checkpoint时的文件写入操作冲突。注释掉该行代码，让Trainer自动处理配置文件的生成与保存。

• 以管理员身份运行训练
  Windows系统下，右键点击终端或IDE选择“以管理员身份运行”，排除系统级别的权限限制。

内容的提问来源于stack exchange，提问作者Vinay Sharma