Spacy Entity Ruler中复杂正则表达式无法匹配问题排查

尝试通过Spacy的Entity Ruler传入正则识别实体，但Spacy无法匹配到目标内容，不过该正则在测试工具中验证有效。

代码示例 #

主代码：

import model_training
import spacy

nlp = spacy.load('en_core_web_trf')
nlp.add_pipe("spacytextblob")

nlp = model_training.train_model_with_regex(nlp)

model_training.py代码：

def train_model_with_regex(nlp):
    ruler = nlp.add_pipe("entity_ruler", before="ner")
    patterns = [
        {
            "label": "VOLUME",
            "pattern": [{"LOWER": {'REGEX": "(?:\d+\s(?:million|hundred|thousand|billion)*\s*)+"}}]
        }
    ]

    ruler.add_patterns(patterns)
    return nlp

期望效果 #

对于文本text = "I have spent 5 million to buy house and 70 thousand for the furniture"，识别出两个VOLUME实体：'5 million'和'70 thousand'，输出格式如下：

{'result': [
    {'label': 'VOLUME', 'text': '5 million'},
    {'label': 'VOLUME', 'text': '70 thousand'}
]}

问题分析与修复 #

核心错误点 #

1. 引号语法错误：代码中{'REGEX": "(?:\d+\s...)"}的引号配对错误，应该修正为{"REGEX": "(?:\d+\s...)"}。
2. Token匹配逻辑错误：当前pattern试图用单个token的LOWER属性匹配正则，但目标实体5 million是两个独立的token（5和million），单个token的属性无法覆盖多token内容，导致正则永远匹配不到。

修复方案 #

方案1：多Token序列匹配（推荐）

针对数字+单位的固定结构，明确匹配数字token和单位token：

def train_model_with_regex(nlp):
    ruler = nlp.add_pipe("entity_ruler", before="ner")
    patterns = [
        {
            "label": "VOLUME",
            "pattern": [
                {"LIKE_NUM": True},
                {"LOWER": {"IN": ["million", "hundred", "thousand", "billion"]}}
            ]
        }
    ]
    ruler.add_patterns(patterns)
    return nlp

方案2：全局文本正则匹配

如果需要更灵活的正则规则，可以用TEXT结合正则，并启用贪婪匹配：

def train_model_with_regex(nlp):
    ruler = nlp.add_pipe("entity_ruler", before="ner")
    patterns = [
        {
            "label": "VOLUME",
            "pattern": [{"TEXT": {"REGEX": r"\d+\s+(million|hundred|thousand|billion)"}}],
            "greedy": "LONGEST"
        }
    ]
    ruler.add_patterns(patterns)
    return nlp

测试验证 #

运行以下代码即可得到期望输出：

text = "I have spent 5 million to buy house and 70 thousand for the furniture"
doc = nlp(text)
result = [{"label": ent.label_, "text": ent.text} for ent in doc.ents if ent.label_ == "VOLUME"]
print({'result': result})

内容的提问来源于stack exchange，提问作者Kamal