使用nlpaug句子增强工具时遇GPT2/XLNet相关错误

错误1：ValueError: 50256 is not in list #

该错误源于GPT2的<|endoftext|> token（ID为50256）未被nlpaug的内部token列表识别，解决步骤如下：

• 手动将GPT2的eos_token设为pad_token，确保tokenizer能正确处理特殊token
• 初始化增强器时传入配置好的tokenizer

示例代码：

from nlpaug.augmenter.sentence import ContextualWordEmbsForSentenceAug
from transformers import GPT2Tokenizer, GPT2LMHeadModel
import torch

# 加载模型与tokenizer
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')

# 配置tokenizer特殊token
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# 初始化句子增强器
aug = ContextualWordEmbsForSentenceAug(
    model_path='gpt2',
    model=model,
    tokenizer=tokenizer,
    device='cuda' if torch.cuda.is_available() else 'cpu',
    action="insert"
)

# 执行增强
original_text = "Your input sentence here"
augmented_text = aug.augment(original_text)
print(augmented_text)

错误2：AttributeError: 'Gpt2' object has no attribute 'MASK_TOKEN' #

GPT2是自回归模型，本身没有MASK_TOKEN（XLNet具备该token），nlpaug默认尝试访问该属性导致报错，解决方法：

• 初始化GPT2增强器时，手动将mask_token指定为GPT2的eos_token

修正后的GPT2增强器初始化代码：

aug = ContextualWordEmbsForSentenceAug(
    model_path='gpt2',
    model=model,
    tokenizer=tokenizer,
    mask_token=tokenizer.eos_token,
    device='cuda' if torch.cuda.is_available() else 'cpu',
    action="insert"
)

XLNet的正确使用示例 #

XLNet原生支持MASK机制，无需额外指定mask_token，直接初始化即可：

from transformers import XLNetTokenizer, XLNetLMHeadModel

tokenizer_xlnet = XLNetTokenizer.from_pretrained('xlnet-base-cased')
model_xlnet = XLNetLMHeadModel.from_pretrained('xlnet-base-cased')

aug_xlnet = ContextualWordEmbsForSentenceAug(
    model_path='xlnet-base-cased',
    model=model_xlnet,
    tokenizer=tokenizer_xlnet,
    device='cuda' if torch.cuda.is_available() else 'cpu',
    action="insert"
)

augmented_xlnet = aug_xlnet.augment(original_text)
print(augmented_xlnet)

关键注意事项 #

• GPT2更适合action="insert"，自回归特性决定其不擅长substitute操作
• Colab环境需保证依赖版本兼容，推荐使用transformers4.28.0、nlpaug1.1.11

内容的提问来源于stack exchange，提问作者Isaac Yves