IndoBERT微调NER任务报错ValueError：模型未返回loss求解决

尝试用自定义数据集微调IndoBERT做命名实体识别（NER），之前用BERT-base-uncased能正常运行，但切换到IndoBERT后调用trainer.train()时出现如下报错：

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[367], line 1
----> 1 trainer.train()

File ~\AppData\Local\anaconda3\Lib\site-packages\transformers\trainer.py:1539, in Trainer.train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
   1537         hf_hub_utils.enable_progress_bars()
   1538 else:
-> 1539     return inner_training_loop(
   1540         args=args,
   1541         resume_from_checkpoint=resume_from_checkpoint,
   1542         trial=trial,
   1543         ignore_keys_for_eval=ignore_keys_for_eval,
   1544     )

File ~\AppData\Local\anaconda3\Lib\site-packages\transformers\trainer.py:1869, in Trainer._inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)
   1866     self.control = self.callback_handler.on_step_begin(args, self.state, self.control)
   1868 with self.accelerator.accumulate(model):
-> 1869     tr_loss_step = self.training_step(model, inputs)
   1871 if (
   1872     args.logging_nan_inf_filter
   1873     and not is_torch_tpu_available()
   1874     and (torch.isnan(tr_loss_step) or torch.isinf(tr_loss_step))
   1875 ):
   1876     # if loss is nan or inf simply add the average of previous logged losses
   1877     tr_loss += tr_loss / (1 + self.state.global_step - self._globalstep_last_logged)

File ~\AppData\Local\anaconda3\Lib\site-packages\transformers\trainer.py:2772, in Trainer.training_step(self, model, inputs)
   2769     return loss_mb.reduce_mean().detach().to(self.args.device)
   2771 with self.compute_loss_context_manager():
-> 2772     loss = self.compute_loss(model, inputs)
   2774 if self.args.n_gpu > 1:
   2775     loss = loss.mean()  # mean() to average on multi-gpu parallel training

File ~\AppData\Local\anaconda3\Lib\site-packages\transformers\trainer.py:2813, in Trainer.compute_loss(self, model, inputs, return_outputs)
   2811 else:
   2812     if isinstance(outputs, dict) and "loss" not in outputs:
-> 2813         raise ValueError(
   2814             "The model did not return a loss from the inputs, only the following keys: "
   2815             f"{','.join(outputs.keys())}. For reference, the inputs it received are {','.join(inputs.keys())}."
   2816         )
   2817     # We don't use .loss here since the model may return tuples instead of ModelOutput.
   2818     loss = outputs["loss"] if isinstance(outputs, dict) else outputs[0]

ValueError: The model did not return a loss from the inputs, only the following keys: last_hidden_state,pooler_output. For reference, the inputs it received are input_ids,token_type_ids,attention_mask.

相关代码 #

Tokenizer初始化

tokenizer = AutoTokenizer.from_pretrained("indolem/indobert-base-uncased")

分词与标签对齐函数

def tokenize_and_align_labels(examples, label_all_tokens=True):
    """
    该函数用于对文本进行分词并将标签与分词后的token对齐，专为命名实体识别（NER）任务设计，因为分词后需要对齐标签。

    参数:
    examples (dict): 包含tokens和对应NER标签的字典。
                     - "tokens": 句子中的单词列表。
                     - "ner_tags": 每个单词对应的实体标签列表。

    label_all_tokens (bool): 标记是否为所有token分配标签的标志。
                             如果为False，仅单词的第一个token会被分配标签，
                             同一单词对应的其他子词token会被分配-100。

    返回:
    tokenized_inputs (dict): 包含分词后的输入以及与token对齐的对应标签的字典。
    """
    tokenized_inputs = tokenizer(examples["text"], truncation=True, is_split_into_words=True)
    labels = []
    for i, label in enumerate(examples["labels"]):
        word_ids = tokenized_inputs.word_ids(batch_index=i)
        # word_ids() => 返回一个列表，将token映射到初始句子中的实际单词
        # 它返回一个指示每个token对应哪个单词的列表
        previous_word_idx = None
        label_ids = []
        # 像`<s>`和`</s>`这样的特殊标记最初映射为None
        # 我们需要将它们的标签设置为-100，以便在损失函数中自动忽略
        for word_idx in word_ids:
            if word_idx is None:
                # 为这些特殊标记设置-100作为标签
                label_ids.append(-100)
            # 对于单词中的其他token，我们根据label_all_tokens标志将标签设置为当前标签或-100
            elif word_idx != previous_word_idx:
                # 如果当前word_idx不等于之前的，这是最常见的情况
                # 添加对应的标签
                label_ids.append(label[word_idx])
            else:
                # 处理具有相同word_idx的子词
                # 如果label_all_tokens为False，也将它们的标签设置为-100
                label_ids.append(label[word_idx] if label_all_tokens else -100)
                # 屏蔽第一个子词之后的子词表示

            previous_word_idx = word_idx
        labels.append(label_ids)
    tokenized_inputs["labels"] = labels
    return tokenized_inputs

模型初始化

model = AutoModel.from_pretrained("indolem/indobert-base-uncased",num_labels=7)

训练参数与Trainer设置

from transformers import TrainingArguments, Trainer
args = TrainingArguments(
"test-ner",
evaluation_strategy = "epoch",
learning_rate=2e-2,
per_device_train_batch_size=4,
per_device_eval_batch_size=4,
num_train_epochs=1,
weight_decay=0.1,
)

def compute_metrics(eval_preds):
    """
    该函数用于计算命名实体识别（NER）任务的评估指标，包括精确率、召回率、F1分数和准确率。

    参数:
    eval_preds (tuple): 包含预测logits和真实标签的元组。

    返回:
    包含精确率、召回率、F1分数和准确率的字典。
    """
    pred_logits, labels = eval_preds

    pred_logits = np.argmax(pred_logits, axis=2)
    # logits和概率的顺序相同，因此不需要应用softmax

    # 移除所有标签为-100的值
    predictions = [
        [label_list[eval_preds] for (eval_preds, l) in zip(prediction, label) if l != -100]
        for prediction, label in zip(pred_logits, labels)
    ]

    true_labels = [
      [label_list[l] for (eval_preds, l) in zip(prediction, label) if l != -100]
       for prediction, label in zip(pred_logits, labels)
   ]
    results = metric.compute(predictions=predictions, references=true_labels)
    return {
   "precision": results["overall_precision"],
   "recall": results["overall_recall"],
   "f1": results["overall_f1"],
  "accuracy": results["overall_accuracy"],
  }

trainer = Trainer(
    model=model,
    args=args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["valid"],
    data_collator=data_collator,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics,
)

trainer.train()

数据集结构 #

DatasetDict({
    train: Dataset({
        features: ['text', 'labels'],
        num_rows: 91
    })
    test: Dataset({
        features: ['text', 'labels'],
        num_rows: 12
    })
    valid: Dataset({
        features: ['text', 'labels'],
        num_rows: 11
    })
})

解决方案 #

1. 核心问题修复：使用正确的模型类 #

初始化模型时用了AutoModel，这是基础BERT模型，仅返回隐藏层输出和池化输出，不会自动计算NER任务的损失。NER属于token级分类任务，必须使用专门的AutoModelForTokenClassification类，它会在BERT基础上添加分类头，并自动处理损失计算。

修改模型初始化代码：

from transformers import AutoModelForTokenClassification

model = AutoModelForTokenClassification.from_pretrained("indolem/indobert-base-uncased", num_labels=7)

2. 额外优化建议 #

• 调整学习率：当前设置的learning_rate=2e-2过大，BERT类模型微调的合理学习率范围是2e-5到5e-5，过高的学习率会导致模型训练不稳定甚至发散。
• 验证标签映射：确认label_list与数据集的标签索引完全对应，避免出现索引越界问题。
• 检查输入格式：确保examples["text"]是按单词拆分的列表（因为使用了is_split_into_words=True），如果是原始字符串需要先做分词拆分。

内容的提问来源于stack exchange，提问作者Alwan Rahmana Subian