使用Transformers做文本复述出现FutureWarning报错如何修复？

问题修复方案 #

触发警告的核心原因是你错误将输入序列的分词逻辑放到了as_target_tokenizer上下文管理器中，该上下文仅用于处理生成阶段的目标文本，输入序列的分词不需要包裹该上下文，调整后即可消除警告。

修复后完整代码 #

import torch
from transformers import PegasusForConditionalGeneration, PegasusTokenizer

model_name = 'tuner007/pegasus_paraphrase'
torch_device = 'cuda' if torch.cuda.is_available() else 'cpu'
tokenizer = PegasusTokenizer.from_pretrained(model_name)
model = PegasusForConditionalGeneration.from_pretrained(model_name).to(torch_device)

def get_response(input_text,num_return_sequences):
    # 直接用分词器__call__方法处理输入，替代废弃的prepare_seq2seq_batch
    tokenized_text = tokenizer([input_text], truncation=True, padding='longest', max_length=60, return_tensors="pt")
    batch = tokenized_text.to(torch_device)
    translated = model.generate(**batch,max_length=60,num_beams=10, num_return_sequences=num_return_sequences, temperature=1.5)
    tgt_text = tokenizer.batch_decode(translated, skip_special_tokens=True)
    return tgt_text

text = "In this video, I will be showing you how to build a stock price web application in Python using the Streamlit and yfinance library."
get_response(text, 5)

改动说明 #

• 移除了包裹输入分词逻辑的as_target_tokenizer上下文，输入序列直接调用分词器处理即可，完全兼容Transformers v4+及后续v5版本的规范
• 补全了分词器输入的列表包裹，符合单条文本输入的处理逻辑，避免输出结果异常

内容的提问来源于stack exchange，提问作者user17142920