PhraseMatcher无法匹配LEMMA的问题咨询——如何实现同时基于LEMMA与LOWER属性匹配

Let's break down why your PhraseMatcher isn't catching the "your" match and fix it step by step:

1. Pipeline Order is Wrong (Critical!) #

You added your _lower_case_lemmas pipe after the "tagger", but spaCy's lemmatizer runs after the tagger, parser, and attribute ruler in the default English pipeline. This means your function is modifying token lemmas before the lemmatizer has even set the correct lemma values (like turning "your" into "-pron-").

Change the pipe placement to run after the lemmatizer so you're working with finalized lemma data:

nlp.add_pipe(self._lower_case_lemmas, after="lemmatizer")

2. Variable Name Typo #

You initialize the matcher as pharsematcher=PhraseMatcher(...) but add rules to self._pharsematcher. Make sure these reference the same object:

# Initialize the matcher as an instance variable
self._pharsematcher = PhraseMatcher(self._nlp.vocab, attr='LEMMA')

3. Verify Pattern & Document Lemma Consistency #

Double-check that both your pattern tokens and input document tokens have matching lemma values. Add debug prints to confirm:

# Check input doc lemmas
doc = self._nlp("Doctor: what is your date of birth")
for token in doc:
    print(f"Text: {token.text}, Lemma: {token.lemma_}")

# Check pattern lemmas
Referal_match_wrods=['your','ur']
your_patern=[self._nlp(text.lower()) for text in Referal_match_wrods]
for pattern in your_patern:
    print(f"Pattern Text: {pattern[0].text}, Pattern Lemma: {pattern[0].lemma_}")

Both should show "your" for the "your" token once the pipeline order is fixed.

4. Ensure AssignDocToPharsMatcher Works Correctly #

Make sure this method returns the matcher's results on the processed document. It should look something like:

def AssignDocToPharsMatcher(self, doc):
    return self._pharsematcher(doc)

Final Check #

After applying these fixes, run your matching code again. The "your" token should now be detected since both the pattern and input document will have identical lowercase lemma values that the PhraseMatcher compares.

内容的提问来源于stack exchange，提问作者Nir Elbaz