如何通过HuggingFace Transformers Trainer获取验证准确率及结果解读

用户代码示例 #

# define the metrics

import numpy as np
from sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score, classification_report

def compute_metrics(p):
    pred, labels = p
    pred = np.argmax(pred, axis=1)

    report = classification_report(labels, pred, digits=4)
    acc = accuracy_score(y_true=labels, y_pred=pred)
    rec = recall_score(y_true=labels, y_pred=pred, average='micro')
    prec = precision_score(y_true=labels, y_pred=pred, average='micro')
    f1 = f1_score(y_true=labels, y_pred=pred, average='micro')

    print("Classification Report:\n{}".format(report))
    return {"accuracy": acc, "precision": prec, "recall": rec, "f1": f1}

from transformers import TrainingArguments, Trainer
from transformers import AutoModelForSequenceClassification

# Get the base model and configuration
model = AutoModelForSequenceClassification.from_pretrained(
    model_chkpt,
    num_labels=7,
    id2label=id2label,
    label2id=label2id,
    ignore_mismatched_sizes=True
)

output_dir = "indonesian-emotion-distilbert-base-cased-finetuned" # name directory
training_args = TrainingArguments(
        output_dir=output_dir,
        evaluation_strategy="epoch",
        save_strategy="epoch",
        learning_rate=1e-5,
        per_device_train_batch_size=16,
        per_device_eval_batch_size=16,
        num_train_epochs=5,
        warmup_ratio=0.01,
        weight_decay=0.01,
        load_best_model_at_end=True,
        metric_for_best_model="accuracy",
        push_to_hub=True)

trainer = Trainer(
    model=model,
    args=training_args,
    tokenizer=tokenizer,
    train_dataset=tokenized_train_data,
    eval_dataset=tokenized_test_data,
    compute_metrics=compute_metrics,
)

trainer.train()

疑问解答 #

1. 结果中的Accuracy是训练准确率还是验证准确率？ #

是验证准确率。原因如下：

• 你在TrainingArguments中设置了evaluation_strategy="epoch"，意味着每个epoch结束后，Trainer会自动在eval_dataset（即你传入的tokenized_test_data）上执行评估。
• compute_metrics函数仅在评估阶段被调用，输入的labels和pred都来自验证集，因此返回的accuracy、precision等指标都是验证集上的结果。
• 训练日志里的Training Loss是训练过程中计算的损失，Validation Loss和对应的Accuracy/Precision/Recall/F1是验证集上的结果，二者一一对应。

2. 如何获取每epoch或step的验证准确率？ #

• 每epoch的验证准确率：你当前的设置已经实现了该功能。evaluation_strategy="epoch"会让Trainer在每个epoch结束后自动评估验证集，日志中输出的Accuracy就是对应epoch的验证准确率，无需额外修改。
• 每step的验证准确率：修改TrainingArguments中的参数即可：
  1. 将evaluation_strategy改为"steps"
  2. 添加eval_steps参数，指定每多少个训练步骤执行一次验证（比如eval_steps=100表示每100步评估一次）
     修改后的training_args示例：

  training_args = TrainingArguments(
          output_dir=output_dir,
          evaluation_strategy="steps",
          eval_steps=100,  # 每100步评估一次
          save_strategy="epoch",
          learning_rate=1e-5,
          per_device_train_batch_size=16,
          per_device_eval_batch_size=16,
          num_train_epochs=5,
          warmup_ratio=0.01,
          weight_decay=0.01,
          load_best_model_at_end=True,
          metric_for_best_model="accuracy",
          push_to_hub=True)
  

  这样Trainer就会在指定的step间隔后评估验证集，并输出对应的验证准确率等指标。

内容的提问来源于stack exchange，提问作者Rangga Saputra