寻求Python中实用的基于词典的文本分类库

1. FuzzyWuzzy + 自定义预处理逻辑 #

FuzzyWuzzy专注于字符串模糊匹配，能直接应对你提到的拼写错误场景（比如示例里的Yelow匹配yellow），结合自定义预处理函数可覆盖小写转换、词干提取、停用词移除等需求：

• 预处理逻辑可完全自定义：转小写、移除标点、词干提取、过滤停用词等步骤都能按需调整
• 用fuzz.partial_ratio或fuzz.token_set_ratio计算文本与实体特征的相似度，通过阈值筛选匹配项
• 动态更新词典只需修改entities字典即可

示例代码：

from fuzzywuzzy import fuzz
from nltk.stem import PorterStemmer
import string
from nltk.corpus import stopwords

def preprocess(text):
    # 转小写+移除标点
    text = text.lower().translate(str.maketrans('', '', string.punctuation))
    # 词干提取
    stemmer = PorterStemmer()
    words = [stemmer.stem(word) for word in text.split()]
    # 过滤停用词
    stop_words = set(stopwords.words('english'))
    words = [word for word in words if word not in stop_words]
    return ' '.join(words)

text = "Yesterday, I ate a Yelow-fruit. It was the longest fruit I ever ate."
entities = {"apple": ["pink", "sphere"], "banana": ["yellow", "tasty", "long"]}
processed_text = preprocess(text)

max_score = 0
result = None
for entity, features in entities.items():
    total_score = 0
    for feat in features:
        processed_feat = preprocess(feat)
        score = fuzz.partial_ratio(processed_text, processed_feat)
        total_score += score
    avg_score = total_score / len(features)
    if avg_score > max_score:
        max_score = avg_score
        result = entity

print(result)  # 输出banana

2. spaCy + 规则/语义匹配 #

spaCy的规则匹配与词向量能力可满足更复杂的语义匹配场景：

• 自带成熟预处理：加载模型后可直接完成小写转换、词形还原、停用词过滤
• 模糊匹配：用Matcher结合正则处理拼写变体；用词向量相似度（doc.similarity()）处理同义词、近义词
• 词典语义扩展：利用词向量的most_similar方法，为实体特征自动补充语义相近词汇

示例代码（简化版）：

import spacy

nlp = spacy.load("en_core_web_md")
text = "Yesterday, I ate a Yelow-fruit. It was the longest fruit I ever ate."
entities = {"apple": ["pink", "sphere"], "banana": ["yellow", "tasty", "long"]}

doc = nlp(text.lower())
# 提取有效词元（移除停用词、标点）
text_tokens = [token.lemma_ for token in doc if not token.is_stop and not token.is_punct]
text_doc = nlp(' '.join(text_tokens))

max_similarity = 0
result = None
for entity, features in entities.items():
    total_sim = 0
    for feat in features:
        feat_doc = nlp(feat.lower())
        sim = text_doc.similarity(feat_doc)
        total_sim += sim
    avg_sim = total_sim / len(features)
    if avg_sim > max_similarity:
        max_similarity = avg_sim
        result = entity

print(result)  # 输出banana

3. Gensim + 语义扩展匹配 #

如果侧重语义相似度与词典自动扩展，Gensim的FastText/Word2Vec模型是合适选择：

• FastText对拼写错误鲁棒性更强（基于字符n-gram训练）
• 用预训练词向量计算文本与实体特征的语义相似度
• 可通过most_similar方法自动为实体特征扩展相似词，丰富词典内容

总结 #

• 偏向模糊字符串匹配+灵活预处理：优先选FuzzyWuzzy配合自定义逻辑
• 需要语义相似度+词典语义扩展：spaCy或Gensim更适配
  以上方案均支持动态更新实体词典，完全匹配你的使用场景。

内容的提问来源于stack exchange，提问作者Neroksi