如何将AllenNLP共指消解模型的分词器替换为空白分词器

你之前的方案失效是因为直接修改predictor._tokenizer私有属性不会生效，共指消解Predictor的文本预处理逻辑绑定在内部_dataset_reader的分词器上，修改该位置的分词器才会真正作用于文本拆分流程。

review = """Judging from previous posts this used to be a good place, but not any longer.
        We, there were four of us, arrived at noon - the place was empty - 
        and the staff acted like we were imposing on them and they were very rude. 
        They never brought us complimentary noodles, ignored repeated requests for sugar, and threw our dishes on the table.
        The food was lousy - too sweet or too salty and the portions tiny.
        After all that, they complained to me about the small tip.
        Avoid this place!"""

from allennlp.data.tokenizers.whitespace_tokenizer import WhitespaceTokenizer
from allennlp.predictors.predictor import Predictor

# 加载预训练共指消解模型
predictor = Predictor.from_path("https://storage.googleapis.com/allennlp-public-models/coref-spanbert-large-2020.02.27.tar.gz")
# 替换dataset reader的分词器为空白分词器
predictor._dataset_reader._tokenizer = WhitespaceTokenizer()

pred = predictor.predict(document=review)
# 输出结果将符合仅按空格拆分、保留标点和词绑定的要求
print(pred['document'])

你使用的该款预训练共指消解模型训练时采用的是SpaCy分词规则，替换为空白分词器后，带标点的组合token（比如place,、longer.）会被模型判定为未登录词，会导致模型效果出现一定程度的下降。如果需要保证精度，建议基于空白分词的语料对模型做微调适配。

内容的提问来源于stack exchange，提问作者rosamariar