HuggingFace Transformers训练报错：'NoneType'与'int'无法比较

问题描述 #

在使用PyTorch 2.0.1+cu118结合HuggingFace Transformers训练EleutherAI/pythia-6.9b模型时，触发如下报错：

TypeError: '>' not supported between instances of 'NoneType' and 'int'

已尝试切换Transformers版本至4.23.1、4.6.0，问题未解决，当前使用版本为4.32.0.dev0。

完整训练代码 #

from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments, Trainer
from datasets import load_dataset

# 加载tokenizer与模型
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/pythia-6.9b")
model = AutoModelForCausalLM.from_pretrained(
    "EleutherAI/pythia-6.9b",
    torch_dtype="auto",
    device_map="auto"
)

# 加载数据集
dataset = load_dataset("your_target_dataset")

# 数据预处理函数
def preprocess_function(examples):
    return tokenizer(
        examples["text"],
        truncation=True,
        padding="max_length",
        max_length=512
    )

tokenized_dataset = dataset.map(preprocess_function, batched=True)

# 训练参数配置
training_args = TrainingArguments(
    output_dir="./pythia_training_results",
    per_device_train_batch_size=4,
    num_train_epochs=3,
    logging_steps=10,
    save_steps=100,
    fp16=True
)

# 初始化Trainer并启动训练
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"]
)

trainer.train()

报错栈 #

Traceback (most recent call last):
  File "train_pythia.py", line 42, in <module>
    trainer.train()
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 1537, in train
    return inner_training_loop(
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 1808, in inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 2560, in training_step
    loss = self.compute_loss(model, inputs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 2592, in compute_loss
    outputs = model(**inputs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/pythia/modeling_pythia.py", line 782, in forward
    outputs = self.generator(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/pythia/modeling_pythia.py", line 655, in forward
    layer_outputs = decoder_layer(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/pythia/modeling_pythia.py", line 375, in forward
    attn_outputs = self.self_attn(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/pythia/modeling_pythia.py", line 293, in forward
    if past_key_value_length > 0:
TypeError: '>' not supported between instances of 'NoneType' and 'int'

排查与解决方案 #

1. 修复模型加载时的参数缺失 #

pythia-6.9b属于大模型，device_map="auto"加载时需补充内存优化参数，避免部分组件初始化失败为None：

model = AutoModelForCausalLM.from_pretrained(
    "EleutherAI/pythia-6.9b",
    torch_dtype="auto",
    device_map="auto",
    low_cpu_mem_usage=True,  # 新增参数
    trust_remote_code=True   # 必要时添加，确保加载模型自定义代码
)

2. 过滤数据集中的无效样本 #

确保数据集中text字段无空值或空白文本，此类样本会导致tokenizer返回异常数据：

# 加载数据集后添加过滤逻辑
dataset = dataset.filter(lambda x: x["text"] is not None and len(x["text"].strip()) > 0)

3. 替换为稳定版Transformers #

4.32.0.dev0为开发版本，存在未修复的bug，建议切换至PyTorch 2.0.1兼容的稳定版：

pip install transformers==4.31.0

4. 检查TrainingArguments的合理性 #

• 若GPU显存不足，降低per_device_train_batch_size至2或1，或添加gradient_accumulation_steps=2分摊显存压力
• 确认GPU支持FP16训练（Turing架构及以上），否则关闭fp16=True

内容的提问来源于stack exchange，提问作者James K J