You need to enable JavaScript to run this app. 使用TRL库PPOTrainer时遇batch为NoneType报错求助
优惠活动
大模型
产品
解决方案
定价
更多

使用TRL库PPOTrainer时遇batch为NoneType报错求助

PPO训练时NoneType索引错误及后续问题解决

问题现象

代码基本照搬PPOTrainer官方教程,替换自定义奖励模型后运行触发以下错误:

错误1:无法索引NoneType对象

Traceback (most recent call last): File "c:\Users\Server\content\test.py", line 68, in <module> query_tensors = batch["input_ids"] ~~~~~^^^^^^^^^^^^^ TypeError: 'NoneType' object is not subscriptable 

若手动判断batch为None并跳过,第二次迭代会触发新错误:

错误2:局部变量未定义

current_batch = send_to_device(current_batch, self.device, non_blocking=self._non_blocking) ^^^^^^^^^^^^^ UnboundLocalError: cannot access local variable 'current_batch' where it is not associated with a value

问题根源

  1. 数据集格式不规范:原tokenize函数生成的input_ids是Python列表,未转为PyTorch张量,也没有统一序列长度(无截断/填充),导致PPOTrainer的dataloader无法正确构造有效batch,返回None。
  2. PPO配置参数缺失:未指定batch_size等核心参数,默认批处理逻辑异常。
  3. 奖励模型输出解析错误:原代码错误解析了文本分类pipeline的输出格式。

修复方案

1. 修正tokenize函数,统一序列格式

改用tokenizer的__call__方法自动处理截断、填充,并返回张量:

def tokenize(sample):
    # 根据模型显存设置合适的max_length,这里设为128
    tokenized = tokenizer(
        sample["query"],
        max_length=128,
        truncation=True,
        padding="max_length",
        return_tensors="pt"
    )
    # 移除多余的维度,适配数据集格式
    sample["input_ids"] = tokenized["input_ids"].squeeze()
    sample["attention_mask"] = tokenized["attention_mask"].squeeze()
    return sample

2. 将数据集转为PyTorch格式

在tokenize映射后添加格式转换:

dataset = dataset.map(tokenize, batched=False)
# 指定数据集格式为PyTorch,包含必要字段
dataset.set_format(type="torch", columns=["input_ids", "attention_mask", "query"])

3. 完善PPOConfig配置

添加batch_size等必要参数:

config = PPOConfig(
    model_name="gpt2",
    learning_rate=1.41e-5,
    batch_size=4,  # 根据自身显存调整大小
    gradient_accumulation_steps=1,
)

4. 修正奖励模型输出解析

文本分类pipeline的输出是字典列表,直接提取score字段即可:

# 原错误写法:rewards = [torch.tensor(output[1]["score"]) for output in pipe_outputs]
rewards = [torch.tensor(output["score"]) for output in pipe_outputs]

修复后的完整代码

from datasets import load_dataset
import torch
from trl import PPOConfig, AutoModelForCausalLMWithValueHead, PPOTrainer
from transformers import AutoTokenizer, pipeline
from tqdm import tqdm

# 加载并预处理数据集
dataset = load_dataset("HuggingFaceH4/cherry_picked_prompts", split="train")
dataset = dataset.rename_column("prompt", "query")
dataset = dataset.remove_columns(["meta", "completion"])

# 配置PPO参数
config = PPOConfig(
    model_name="gpt2",
    learning_rate=1.41e-5,
    batch_size=4,
    gradient_accumulation_steps=1,
)

# 加载模型与tokenizer
model = AutoModelForCausalLMWithValueHead.from_pretrained(config.model_name)
tokenizer = AutoTokenizer.from_pretrained(config.model_name)
tokenizer.pad_token = tokenizer.eos_token

# 加载奖励模型
reward_model = pipeline("text-classification", model="lvwerra/distilbert-imdb")

# 修正后的tokenize函数
def tokenize(sample):
    tokenized = tokenizer(
        sample["query"],
        max_length=128,
        truncation=True,
        padding="max_length",
        return_tensors="pt"
    )
    sample["input_ids"] = tokenized["input_ids"].squeeze()
    sample["attention_mask"] = tokenized["attention_mask"].squeeze()
    return sample

# 处理数据集并转为PyTorch格式
dataset = dataset.map(tokenize, batched=False)
dataset.set_format(type="torch", columns=["input_ids", "attention_mask", "query"])

# 初始化PPOTrainer
ppo_trainer = PPOTrainer(
    model=model,
    config=config,
    dataset=dataset,
    tokenizer=tokenizer,
)

# 生成参数配置
generation_kwargs = {
    "min_length": -1,
    "top_k": 0.0,
    "top_p": 1.0,
    "do_sample": True,
    "pad_token_id": tokenizer.eos_token_id,
    "max_new_tokens": 64,  # 新增,避免生成过长文本
}

# 开始训练
epochs = 10
for epoch in tqdm(range(epochs), desc="epoch"):
    for batch in tqdm(ppo_trainer.dataloader, desc="batch"):
        query_tensors = batch["input_ids"]

        # 生成模型回复
        response_tensors = ppo_trainer.generate(query_tensors, **generation_kwargs)
        batch["response"] = [tokenizer.decode(r.squeeze()) for r in response_tensors]

        # 计算奖励分数
        texts = [q + r for q, r in zip(batch["query"], batch["response"])]
        pipe_outputs = reward_model(texts)
        rewards = [torch.tensor(output["score"]) for output in pipe_outputs]

        # 执行PPO训练步骤
        stats = ppo_trainer.step(query_tensors, response_tensors, rewards)
        ppo_trainer.log_stats(stats, batch, rewards)

# 保存模型
ppo_trainer.save_pretrained("my_ppo_model")

内容的提问来源于stack exchange,提问作者KyTDK

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级,最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分), 限时 9.9 元起

最近更新时间:2026.06.16 20:24:53