Helsinki模型基于Huggingface Trainer微调时评估性能骤降问题

我在AWS SageMaker Notebook中，参照Huggingface官方文档，用Trainer对Helsinki-NLP/opus-mt-de-fr模型做德译法微调，采用5折交叉验证验证性能。训练阶段速度能达到2.7 iter/s，但评估阶段性能骤降到0.07 iter/s。

微调代码 #

#checkpoint
checkpoint = "Helsinki-NLP/opus-mt-de-fr"
source_lang = "de"
target_lang = "fr"
prefix = "Übersetzen Deutsch ins Französisch: "

#dataset
dataset_opus100 = load_dataset("opus100", "de-fr", split="test")
final_corpus_dataset= dataset_opus100 
# preprocess
def preprocess_function(examples):
    inputs = [prefix + example[source_lang] for example in examples["translation"]]
    targets = [example[target_lang] for example in examples["translation"]]
    model_inputs = tokenizer(
        inputs, text_target=targets, max_length=128, truncation=True
    )
    return model_inputs
# metric
def postprocess_text(preds, labels):
    preds = [pred.strip() for pred in preds]
    labels = [[label.strip()] for label in labels]

    return preds, labels


def compute_metrics(eval_preds):
    preds, labels = eval_preds
    if isinstance(preds, tuple):
        preds = preds[0]
    decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)

    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

    decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)

    result = metric.compute(predictions=decoded_preds, references=decoded_labels)
    result = {"bleu": result["score"]}

    prediction_lens = [
        np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds
    ]
    result["gen_len"] = np.mean(prediction_lens)
    result = {k: round(v, 4) for k, v in result.items()}
    # print(result)
    return result

# config_params
exp_name = "de-fr-helsinki"


seed = 42
kfold = KFold(n_splits=5, shuffle=True, random_state=seed)
# K-fold Cross Validation model evaluation
for fold, (train_ids, val_ids) in enumerate(kfold.split(final_corpus_dataset)):
    # if fold==0 or fold==1:
    #     continue
    # metric
    metric = evaluate.load("sacrebleu")
    # arguments for training
    training_args = Seq2SeqTrainingArguments(
        output_dir=f"test/huggingface_exps/{exp_name}/fold{fold}",
        evaluation_strategy="epoch",
        logging_strategy="epoch",
        # logging_steps=4,
        learning_rate=2e-5,
        per_device_train_batch_size=64,
        per_device_eval_batch_size=64,
        weight_decay=0.01,
        num_train_epochs=10,
        save_total_limit=2,
        save_strategy="epoch",
        load_best_model_at_end=True,
        predict_with_generate=True,
        fp16=False,
        push_to_hub=False,
        # tensorboard log directory
        logging_dir=f"test/huggingface_exps/{exp_name}/fold{fold}/runs",
        report_to=["tensorboard"],
        # dataloader_num_workers=2,
    )
    print("fold", fold)
    # reset tokenizer and model
    tokenizer = AutoTokenizer.from_pretrained(checkpoint)
    data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=checkpoint)
    model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)
    # select splits
    train_dataset = final_corpus_dataset.select(train_ids)
    eval_dataset = final_corpus_dataset.select(val_ids)

    # preprocess
    train_dataset = train_dataset.map(preprocess_function, batched=True)
    eval_dataset = eval_dataset.map(preprocess_function, batched=True)

    trainer = Seq2SeqTrainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        tokenizer=tokenizer,
        data_collator=data_collator,
        compute_metrics=compute_metrics,
        callbacks=[
            CombinedTensorBoardCallback,
            EarlyStoppingCallback(early_stopping_patience=3),
        ],
    )

    train_result = trainer.train()

    # compute train results
    metrics = train_result.metrics

    # print metrics history
    import os

    # Create the directory if it doesn't exist
    os.makedirs(f"test/huggingface_exps/{exp_name}/fold{fold}/", exist_ok=True)

    with open(f"test/huggingface_exps/{exp_name}/fold{fold}/console.json", "w") as f:
        json.dump(trainer.state.log_history, f)

已排查项 #

• GPU内存充足（占用14.5GB/15.5GB）
• 调整评估批次大小无改善
• 训练时曾遇ValueError: This tokenizer cannot be instantiated. Please make sure you have sentencepiece installed in order to use this tokenizer报错，通过! pip install transformers[sentencepiece]解决，不确定是否影响评估性能

依赖包安装顺序 #

! pip install transformers==4.28.0
! pip install datasets
! pip install evaluate
! pip install torch
! pip install sklearn
! pip install tensorboard
! pip install sacrebleu
! pip install accelerate -U
! pip install transformers[sentencepiece]

解决方案建议 #

1. 开启FP16混合精度 #

当前训练和评估都未启用FP16（fp16=False），GPU在FP32下的生成效率远低于FP16。修改训练参数：

fp16=True

SageMaker GPU原生支持FP16，开启后能显著提升评估阶段的生成速度。

2. 优化生成策略参数 #

评估时predict_with_generate=True会触发beam搜索，默认num_beams=5会大幅拖慢速度。调整生成参数：

generation_num_beams=2,  # 降低beam数量，平衡速度与性能
generation_max_length=128

减少beam数量对BLEU分数影响有限，但能让生成速度数倍提升。

3. 提前预处理数据集 #

当前每个fold都重复加载、预处理数据集，将预处理步骤移到循环外：

# 循环外执行一次
final_corpus_dataset = load_dataset("opus100", "de-fr", split="test")
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
final_corpus_dataset = final_corpus_dataset.map(preprocess_function, batched=True)

# 循环内直接拆分
train_dataset = final_corpus_dataset.select(train_ids)
eval_dataset = final_corpus_dataset.select(val_ids)

避免重复预处理带来的时间浪费。

4. 启用数据加载多线程 #

开启dataloader_num_workers让数据加载并行，避免GPU等待数据：

dataloader_num_workers=4  # 根据实例CPU核心数调整，推荐4-8

5. 排查评估指标计算耗时 #

sacrebleu的计算可能成为瓶颈，可以单独测试该步骤耗时。如果是指标计算问题：

• 适当缩小验证集规模
• 换用更轻量的BLEU计算实现

内容的提问来源于stack exchange，提问作者sffgsfgs