Transformer.forward()报错：收到意外关键字参数'labels'求助

我在实现Lewis Tunstall所著《NLP with Transformers》第10章内容时，运行以下代码单元遇到错误：

from transformers.optimization import get_scheduler 

from accelerate import Accelerator

set_seed(args.seed)

# Accelerator
accelerator = Accelerator()
samples_per_step = accelerator.state.num_processes * args.train_batch_size

# Logging
#logger, tb_writer, run_name = setup_logging(project_name.split("/")[1])
#logger.info(accelerator.state)

# Load model and tokenizer
#if accelerator.is_main_process:
#hf_repo = Repository("./", clone_from=project_name, revision=run_name)
#model = AutoModelForCausalLM.from_pretrained("./", gradient_checkpointing=True)
#tokenizer = AutoTokenizer.from_pretrained("./")

# Load dataset and dataloader
dataset_name = 'transformersbook/codeparrot'
train_dataloader, eval_dataloader = create_dataloaders(dataset_name)

# Prepare the optimizer and learning rate scheduler
optimizer = torch.optim.AdamW(get_grouped_params(model), lr=args.learning_rate)
lr_scheduler = get_scheduler(name=args.lr_scheduler_type, optimizer=optimizer,
                         num_warmup_steps=args.num_warmup_steps,
                         num_training_steps=args.max_train_steps,)
def get_lr():
    return optimizer.param_groups[0]['lr']

# Prepare everything with our `accelerator` (order of args is not important)
model, optimizer, train_dataloader, eval_dataloader = accelerator.prepare(
    model, optimizer, train_dataloader, eval_dataloader)

# Train model
model.train()
completed_steps = 0
for step, batch in enumerate(train_dataloader, start=1):
    loss = model(batch, labels=batch).loss
    log_metrics(step, {'lr': get_lr(), 'samples': step*samples_per_step,
                       'steps': completed_steps, 'loss/train': loss.item()})
    loss = loss / args.gradient_accumulation_steps
    accelerator.backward(loss)
    if step % args.gradient_accumulation_steps == 0:
        optimizer.step()
        lr_scheduler.step()
        optimizer.zero_grad()
        completed_steps += 1
    if step % args.save_checkpoint_steps == 0:
        logging.info('Evaluating and saving model checkpoint')
        eval_loss, perplexity = evaluate()
        log_metrics(step, {'loss/eval': eval_loss, 'perplexity': perplexity})
        accelerator.wait_for_everyone()
        unwrapped_model = accelerator.unwrap_model(model)
    
        model.train()
    if completed_steps >= args.max_train_steps:
        break

# Evaluate and save the last checkpoint
logging.info('Evaluating and saving model after training')
eval_loss, perplexity = evaluate()
log_metrics(step, {'loss/eval': eval_loss, 'perplexity': perplexity})
accelerator.wait_for_everyone()
unwrapped_model = accelerator.unwrap_model(model)

错误详情：

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[81], line 40
     38 completed_steps = 0
     39 for step, batch in enumerate(train_dataloader, start=1):
---> 40     loss = model(batch, labels=batch).loss
     41     log_metrics(step, {'lr': get_lr(), 'samples': step*samples_per_step,
     42                        'steps': completed_steps, 'loss/train': loss.item()})
     43     loss = loss / args.gradient_accumulation_steps

File d:\Crimson Energy Experts\env\Lib\site-packages\torch\nn\modules\module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

TypeError: Transformer.forward() got an unexpected keyword argument 'labels'

解决方案 #

问题根源

你当前使用的model是基础Transformer类（PyTorch原生或Hugging Face基础版），这类模型的forward方法不支持labels参数。只有专门的因果语言模型（比如GPT系列的AutoModelForCausalLM）才会在forward中接收labels并自动计算损失。

修复步骤

1. 加载正确的模型类
   取消注释代码中加载AutoModelForCausalLM的部分，确保使用因果语言模型：

   # 取消注释以下代码
   if accelerator.is_main_process:
       hf_repo = Repository("./", clone_from=project_name, revision=run_name)
   model = AutoModelForCausalLM.from_pretrained("./", gradient_checkpointing=True)
   tokenizer = AutoTokenizer.from_pretrained("./")
   

   如果是从头初始化模型，改用from_config方法：

   from transformers import AutoConfig
   config = AutoConfig.from_pretrained("gpt2") # 替换为你需要的模型配置
   model = AutoModelForCausalLM.from_config(config)
   

2. 调整模型调用方式
   确保输入格式符合模型要求，因果语言模型通常接受input_ids作为关键字参数：

   # 替换原有的model调用代码
   loss = model(input_ids=batch, labels=batch).loss
   

   如果你的batch是包含多个字段的字典（比如{'input_ids': ..., 'attention_mask': ...}），直接传入字典即可：

   loss = model(**batch, labels=batch['input_ids']).loss
   

3. 手动计算损失（如果坚持用基础Transformer）
   若必须使用基础Transformer类，需要手动用模型输出的logits计算交叉熵损失：

   from torch.nn import CrossEntropyLoss
   
   # 原model调用替换为以下代码
   outputs = model(batch)
   logits = outputs.last_hidden_state
   loss_fct = CrossEntropyLoss()
   # 因果语言建模中，标签是输入的下一个token，所以需要偏移
   loss = loss_fct(logits[:, :-1, :].reshape(-1, logits.size(-1)), batch[:, 1:].reshape(-1))
   

内容的提问来源于stack exchange，提问作者Bhupinder singh