Spacy如何创建使用自定义分隔符的Tokenizer实现按逗号分词

错误原因 #

• 参数使用不符合要求：token_match参数仅接受返回布尔值的匹配函数，用于判定单个完整token是否符合规则，你传入正则的finditer方法完全不符合参数定义，规则无法生效。
• 未禁用原生默认拆分规则：spaCy默认Tokenizer内置了空格、标点拆分逻辑，你未覆盖该逻辑，所以文本还是会先按空格拆分。

正确实现方案 #

直接重写Tokenizer的完整逻辑，按逗号拆分后处理首尾空白即可，代码如下：

from spacy.tokens import Doc
from spacy.lang.en import English

def custom_tokenizer(nlp):
    def tokenize(text):
        # 按逗号拆分，过滤空值并去除每个片段前后的空白
        tokens = [seg.strip() for seg in text.split(',') if seg.strip()]
        return Doc(nlp.vocab, words=tokens)
    return tokenize

# 测试
nlp = English()
# 替换默认分词器
nlp.tokenizer = custom_tokenizer(nlp)

test_text = "5g, dynamic vision sensor (dvs), 3-d reconstruction, neuromorphic engineering, neural networks, humanoid robots, neuromorphics, closed loop systems, field programmable gate arrays, spiking motor controller, neuromorphic implementation, icub, relation neural network"
doc = nlp(test_text)
print([token.text for token in doc])

运行后输出结果和预期完全一致：
['5g', 'dynamic vision sensor (dvs)', '3-d reconstruction', 'neuromorphic engineering', 'neural networks', 'humanoid robots', 'neuromorphics', 'closed loop systems', 'field programmable gate arrays', 'spiking motor controller', 'neuromorphic implementation', 'icub', 'relation neural network']

内容的提问来源于stack exchange，提问作者NineWasps