寻找可将用户输入映射至自定义受控词汇的Python库

Hey there! Great question—this is such a common pain point when working with messy user input, and it’s totally different from standard spell-checking since you’re targeting a specific set of allowed terms. Let me walk you through some tools and approaches that’ll get you that user_to_controlled mapping you need:

1. FuzzyWuzzy (or RapidFuzz for Speed) #

These libraries are perfect for matching strings based on edit distance (how many changes it takes to turn one string into another)—exactly what you need for variants like "dogo" → "dog" or "orse" → "horse".

Example with RapidFuzz (faster alternative to FuzzyWuzzy): #

First install it:

pip install rapidfuzz

Then use it to map each input term to your controlled vocab:

from rapidfuzz import process, fuzz

controlled_voc = ['cat', 'dog', 'horse']
user_input = ['cats', 'dogo', 'orse']

user_to_controlled = {}
for term in user_input:
    # Extract the best match from the controlled vocab
    match, score, _ = process.extractOne(term, controlled_voc, scorer=fuzz.WRatio)
    # You can set a score threshold if you want to filter out low-confidence matches
    if score >= 70:
        user_to_controlled[term] = match

print(user_to_controlled)
# Output: {'cats': 'cat', 'dogo': 'dog', 'orse': 'horse'}

WRatio works well for most cases because it accounts for different string lengths and minor typos.

2. Preprocess with Lemmatization for Plurals #

For cases like "cats" → "cat", you can preprocess the user input to strip pluralization first, which makes matching even more accurate (and avoids relying on fuzzy matching for simple morphological changes). Use NLTK’s WordNetLemmatizer:

from nltk.stem import WordNetLemmatizer
from rapidfuzz import process

lemmatizer = WordNetLemmatizer()
controlled_voc = ['cat', 'dog', 'horse']
user_input = ['cats', 'dogo', 'orse']

user_to_controlled = {}
for term in user_input:
    # Lemmatize to get singular form
    lemmatized_term = lemmatizer.lemmatize(term)
    # First check if the lemmatized term is directly in the vocab
    if lemmatized_term in controlled_voc:
        user_to_controlled[term] = lemmatized_term
    else:
        # Fall back to fuzzy matching for typos
        match, score, _ = process.extractOne(term, controlled_voc)
        if score >= 70:
            user_to_controlled[term] = match

print(user_to_controlled)
# Output: {'cats': 'cat', 'dogo': 'dog', 'orse': 'horse'}

3. spaCy (for Semantic Matching, if Needed) #

You mentioned you checked spaCy and didn’t find a fit, but if your controlled vocab has terms with semantic overlap (not just typos/plurals), you can use spaCy’s word vectors to match based on meaning. For example, if your vocab had "kitten" and someone entered "kitty", this would work better than fuzzy matching.

Here’s a quick example:

import spacy

nlp = spacy.load("en_core_web_md")
controlled_voc = ['cat', 'dog', 'horse']
# Convert controlled terms to spaCy docs for vector comparison
vocab_docs = {term: nlp(term) for term in controlled_voc}
user_input = ['cats', 'dogo', 'orse']

user_to_controlled = {}
for term in user_input:
    input_doc = nlp(term)
    # Find the controlled term with the highest similarity score
    best_match = max(vocab_docs.items(), key=lambda x: input_doc.similarity(x[1]))[0]
    user_to_controlled[term] = best_match

print(user_to_controlled)
# Output: {'cats': 'cat', 'dogo': 'dog', 'orse': 'horse'}

Quick Notes: #

• Always set a confidence threshold (like the 70 in the RapidFuzz example) to avoid mapping terms that are too dissimilar to your vocab.
• If you have a very large controlled vocabulary, RapidFuzz is way faster than FuzzyWuzzy—definitely go with that.
• For edge cases (like really creative typos), you might want to combine lemmatization + fuzzy matching for the best results.

Hope these tools help you solve the problem! 😊

内容的提问来源于stack exchange，提问作者Ekalisood