如何重新还原分词后的email地址，避免@与前后内容被拆分？

问题根因 #

NLTK默认的word_tokenize遵循Treebank分词规范，会将邮箱地址中的@符号单独拆分为独立token，生成mobydick123、@、gmail.com三个独立分片，而TreebankWordDetokenizer默认没有针对邮箱格式的合并规则，因此无法还原为完整邮箱。

方案1：自定义分词规则，避免邮箱被拆分 #

优先用正则匹配识别完整邮箱，再对剩余文本做普通分词，从根源避免邮箱被拆分：

from nltk.tokenize.treebank import TreebankWordDetokenizer
from nltk.tokenize import TweetTokenizer
from itertools import groupby
import re

# 自定义支持邮箱识别的分词器
class EmailFriendlyTokenizer:
    def __init__(self):
        self.email_pattern = re.compile(r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}')
        self.base_tokenizer = TweetTokenizer()
    
    def tokenize(self, text):
        tokens = []
        last_end = 0
        for match in self.email_pattern.finditer(text):
            start, end = match.span()
            if start > last_end:
                tokens.extend(self.base_tokenizer.tokenize(text[last_end:start]))
            tokens.append(match.group())
            last_end = end
        if last_end < len(text):
            tokens.extend(self.base_tokenizer.tokenize(text[last_end:]))
        return tokens

input_sent = 'Herman Melvilles email is mobydick123@gmail.com'
tokenizer = EmailFriendlyTokenizer()
tokens = tokenizer.tokenize(input_sent)
print(tokens)

# 后续原有逻辑不变
myList = list(filter((']').__ne__, tokens))
myList = list(filter(('[').__ne__, myList))
res = [i[0] for i in groupby(myList)]
my_list = list(map(lambda item: item.replace("W_NLP_PERSON", "[W_NLP_PERSON]").replace('W_NLP_DATE', '[W_NLP_DATE]').replace('W_NLP_IMEI_HARDWARE_ID', '[W_NLP_IMEI_HARDWARE_ID]').replace('W_NLP_IP_ADDRESS', '[W_NLP_IP_ADDRESS]'), res))
my_list = TreebankWordDetokenizer().detokenize(my_list)
print(my_list)

方案2：分词后合并邮箱分片 #

如果不想修改原有分词逻辑，可以在分词后扫描token列表，合并符合邮箱结构的分片：

from nltk.tokenize.treebank import TreebankWordDetokenizer
from nltk.tokenize import word_tokenize
from itertools import groupby
import re

input_sent = 'Herman Melvilles email is mobydick123@gmail.com'
tokens = word_tokenize(input_sent)
print(tokens)

# 合并邮箱分片逻辑
merged_tokens = []
i = 0
n = len(tokens)
domain_pattern = re.compile(r'[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}')
while i < n:
    if i+2 < n and tokens[i+1] == '@' and domain_pattern.match(tokens[i+2]):
        merged_tokens.append(f"{tokens[i]}{tokens[i+1]}{tokens[i+2]}")
        i += 3
    else:
        merged_tokens.append(tokens[i])
        i += 1

# 后续原有逻辑改用合并后的token列表
myList = list(filter((']').__ne__, merged_tokens))
myList = list(filter(('[').__ne__, myList))
res = [i[0] for i in groupby(myList)]
my_list = list(map(lambda item: item.replace("W_NLP_PERSON", "[W_NLP_PERSON]").replace('W_NLP_DATE', '[W_NLP_DATE]').replace('W_NLP_IMEI_HARDWARE_ID', '[W_NLP_IMEI_HARDWARE_ID]').replace('W_NLP_IP_ADDRESS', '[W_NLP_IP_ADDRESS]'), res))
my_list = TreebankWordDetokenizer().detokenize(my_list)
print(my_list)

方案3：使用spaCy分词 #

你已导入的spaCy默认分词器本身就支持识别邮箱为独立token，无需额外自定义规则：

from nltk.tokenize.treebank import TreebankWordDetokenizer
from itertools import groupby
import spacy

# 加载轻量英文模型，关闭不需要的管线提升速度
nlp = spacy.load("en_core_web_sm", disable=["parser", "ner", "tagger"])
input_sent = 'Herman Melvilles email is mobydick123@gmail.com'
doc = nlp(input_sent)
tokens = [token.text for token in doc]
print(tokens)

# 后续原有逻辑不变
myList = list(filter((']').__ne__, tokens))
myList = list(filter(('[').__ne__, myList))
res = [i[0] for i in groupby(myList)]
my_list = list(map(lambda item: item.replace("W_NLP_PERSON", "[W_NLP_PERSON]").replace('W_NLP_DATE', '[W_NLP_DATE]').replace('W_NLP_IMEI_HARDWARE_ID', '[W_NLP_IMEI_HARDWARE_ID]').replace('W_NLP_IP_ADDRESS', '[W_NLP_IP_ADDRESS]'), res))
my_list = TreebankWordDetokenizer().detokenize(my_list)
print(my_list)

内容的提问来源于stack exchange，提问作者albusdemens