使用HuggingFace分词时如何为缺失句尾标点的语句自动补全句号？

你只需要在解码完成后对输出字符串做末尾标点校验即可，参考实现如下：

import torch
from transformers import AutoTokenizer

# 定义合法句尾标点集合
END_PUNCTUATION = {'.', '?', '!'}

def append_punc_if_needed(text):
    # 先去除首尾空白字符，避免末尾空格影响判断
    text_stripped = text.strip()
    # 兼容空字符串场景
    if not text_stripped:
        return text
    # 末尾无指定标点则追加句号
    if text_stripped[-1] not in END_PUNCTUATION:
        return f"{text}."
    return text

tokenizer = AutoTokenizer.from_pretrained("vinai/bertweet-base", normalization=True)

line = "DHEC confirms https://postandcourier.com/health/covid19/sc-has-first-two-presumptive-cases-of-coronavirus-dhec-confirms/article_bddfe4ae-5fd3-11ea-9ce4-5f495366cee6.html?utm_medium=social&utm_source=twitter&utm_campaign=user-share… via @postandcourier 😢"
ids = tokenizer.encode(line)
decoded_text = tokenizer.decode(ids, skip_special_tokens=True)
processed_text = append_punc_if_needed(decoded_text)

print(processed_text)

运行后输出即为需求的效果：'DHEC confirms HTTPURL... via @USER :crying_face:.'，如果原解码结果末尾已经存在?/!/.中的任意一种，函数会直接返回原内容，不会重复添加标点。

内容的提问来源于stack exchange，提问作者DogEatDog