使用TRL库PPOTrainer时遇batch为NoneType报错求助
PPO训练时NoneType索引错误及后续问题解决
问题现象
代码基本照搬PPOTrainer官方教程,替换自定义奖励模型后运行触发以下错误:
错误1:无法索引NoneType对象
Traceback (most recent call last): File "c:\Users\Server\content\test.py", line 68, in <module> query_tensors = batch["input_ids"] ~~~~~^^^^^^^^^^^^^ TypeError: 'NoneType' object is not subscriptable
若手动判断batch为None并跳过,第二次迭代会触发新错误:
错误2:局部变量未定义
current_batch = send_to_device(current_batch, self.device, non_blocking=self._non_blocking) ^^^^^^^^^^^^^ UnboundLocalError: cannot access local variable 'current_batch' where it is not associated with a value
问题根源
- 数据集格式不规范:原tokenize函数生成的
input_ids是Python列表,未转为PyTorch张量,也没有统一序列长度(无截断/填充),导致PPOTrainer的dataloader无法正确构造有效batch,返回None。 - PPO配置参数缺失:未指定
batch_size等核心参数,默认批处理逻辑异常。 - 奖励模型输出解析错误:原代码错误解析了文本分类pipeline的输出格式。
修复方案
1. 修正tokenize函数,统一序列格式
改用tokenizer的__call__方法自动处理截断、填充,并返回张量:
def tokenize(sample): # 根据模型显存设置合适的max_length,这里设为128 tokenized = tokenizer( sample["query"], max_length=128, truncation=True, padding="max_length", return_tensors="pt" ) # 移除多余的维度,适配数据集格式 sample["input_ids"] = tokenized["input_ids"].squeeze() sample["attention_mask"] = tokenized["attention_mask"].squeeze() return sample
2. 将数据集转为PyTorch格式
在tokenize映射后添加格式转换:
dataset = dataset.map(tokenize, batched=False) # 指定数据集格式为PyTorch,包含必要字段 dataset.set_format(type="torch", columns=["input_ids", "attention_mask", "query"])
3. 完善PPOConfig配置
添加batch_size等必要参数:
config = PPOConfig( model_name="gpt2", learning_rate=1.41e-5, batch_size=4, # 根据自身显存调整大小 gradient_accumulation_steps=1, )
4. 修正奖励模型输出解析
文本分类pipeline的输出是字典列表,直接提取score字段即可:
# 原错误写法:rewards = [torch.tensor(output[1]["score"]) for output in pipe_outputs] rewards = [torch.tensor(output["score"]) for output in pipe_outputs]
修复后的完整代码
from datasets import load_dataset import torch from trl import PPOConfig, AutoModelForCausalLMWithValueHead, PPOTrainer from transformers import AutoTokenizer, pipeline from tqdm import tqdm # 加载并预处理数据集 dataset = load_dataset("HuggingFaceH4/cherry_picked_prompts", split="train") dataset = dataset.rename_column("prompt", "query") dataset = dataset.remove_columns(["meta", "completion"]) # 配置PPO参数 config = PPOConfig( model_name="gpt2", learning_rate=1.41e-5, batch_size=4, gradient_accumulation_steps=1, ) # 加载模型与tokenizer model = AutoModelForCausalLMWithValueHead.from_pretrained(config.model_name) tokenizer = AutoTokenizer.from_pretrained(config.model_name) tokenizer.pad_token = tokenizer.eos_token # 加载奖励模型 reward_model = pipeline("text-classification", model="lvwerra/distilbert-imdb") # 修正后的tokenize函数 def tokenize(sample): tokenized = tokenizer( sample["query"], max_length=128, truncation=True, padding="max_length", return_tensors="pt" ) sample["input_ids"] = tokenized["input_ids"].squeeze() sample["attention_mask"] = tokenized["attention_mask"].squeeze() return sample # 处理数据集并转为PyTorch格式 dataset = dataset.map(tokenize, batched=False) dataset.set_format(type="torch", columns=["input_ids", "attention_mask", "query"]) # 初始化PPOTrainer ppo_trainer = PPOTrainer( model=model, config=config, dataset=dataset, tokenizer=tokenizer, ) # 生成参数配置 generation_kwargs = { "min_length": -1, "top_k": 0.0, "top_p": 1.0, "do_sample": True, "pad_token_id": tokenizer.eos_token_id, "max_new_tokens": 64, # 新增,避免生成过长文本 } # 开始训练 epochs = 10 for epoch in tqdm(range(epochs), desc="epoch"): for batch in tqdm(ppo_trainer.dataloader, desc="batch"): query_tensors = batch["input_ids"] # 生成模型回复 response_tensors = ppo_trainer.generate(query_tensors, **generation_kwargs) batch["response"] = [tokenizer.decode(r.squeeze()) for r in response_tensors] # 计算奖励分数 texts = [q + r for q, r in zip(batch["query"], batch["response"])] pipe_outputs = reward_model(texts) rewards = [torch.tensor(output["score"]) for output in pipe_outputs] # 执行PPO训练步骤 stats = ppo_trainer.step(query_tensors, response_tensors, rewards) ppo_trainer.log_stats(stats, batch, rewards) # 保存模型 ppo_trainer.save_pretrained("my_ppo_model")
内容的提问来源于stack exchange,提问作者KyTDK
相关产品推荐
相关产品推荐

