求助：R/Python中可识别实体的上下文感知模糊匹配算法

针对加权模糊匹配因低频配料词干扰导致实体匹配错误的问题，以下提供R和Python两种环境下的解决方案，核心思路是优先识别食品主实体（如bagel、hamburger）或利用语义嵌入捕捉上下文，避免配料词的权重干扰。

R语言方案：核心实体提取+分层匹配 #

通过词性标注提取字符串中的核心食品实体，先做实体精确匹配，未匹配的再进行模糊匹配，确保主食材优先：

library(udpipe)
library(dplyr)
library(fedmatch)

# 下载并加载英语词性标注模型
udmodel <- udpipe_download_model(language = "english")
udmodel <- udpipe_load_model(udmodel$file_model)

# 定义核心食品提取函数：提取字符串中第一个名词/专有名词作为主食材
extract_core_food <- function(text) {
  parsed <- udpipe_annotate(udmodel, x = text) %>% as.data.frame()
  core_nouns <- parsed %>% 
    filter(upos %in% c("NOUN", "PROPN")) %>% 
    head(1) %>% 
    pull(token)
  if(length(core_nouns) == 0) return(text)
  return(core_nouns)
}

# 准备原始数据
foods1 <- c('bagel plain','bagel with raisins and olives', 'hamburger','bagel with olives','bagel with raisins')
foods1_id <- seq.int(1,length(foods1))
foods2 <- c('bagel','pizza','salad with raisins','tuna and olives')
foods2_id <- c(letters[1:length(foods2)])

foods1_df <- data.frame(foods1_id, foods1, stringsAsFactors = F) %>%
  mutate(core_food = sapply(foods1, extract_core_food))
foods2_df <- data.frame(foods2_id, foods2, stringsAsFactors = F) %>%
  mutate(core_food = sapply(foods2, extract_core_food))

# 1. 核心实体精确匹配
exact_matches <- foods1_df %>%
  inner_join(foods2_df, by = "core_food") %>%
  select(foods1_id, foods2_id, foods1, foods2)

# 2. 提取未匹配记录，进行模糊匹配
unmatched1 <- foods1_df %>% filter(!foods1_id %in% exact_matches$foods1_id)
unmatched2 <- foods2_df %>% filter(!foods2_id %in% exact_matches$foods2_id)

fuzzy_unmatched <- merge_plus(data1 = unmatched1,
                              data2 = unmatched2,
                              by.x = "foods1",
                              by.y = "foods2",
                              match_type = "fuzzy",
                              fuzzy_settings = build_fuzzy_settings(method = "wgt_jaccard", nthread = 2, maxDist = .75),
                              unique_key_1 = "foods1_id",
                              unique_key_2 = "foods2_id")

# 合并最终结果
final_matches <- bind_rows(exact_matches, fuzzy_unmatched$matches)
print(final_matches)

该方案会先将所有含"bagel"的记录匹配到foods2中的"bagel"，避免低频词"raisins"干扰。

Python语言方案：两种实现路径 #

路径1：语义嵌入匹配（上下文感知） #

利用预训练的句子嵌入模型计算语义相似度，捕捉字符串的整体上下文，主食材的语义权重会自然高于配料：

from sentence_transformers import SentenceTransformer, util
import pandas as pd

# 加载轻量级预训练模型
model = SentenceTransformer('all-MiniLM-L6-v2')

# 原始数据
foods1 = ['bagel plain','bagel with raisins and olives', 'hamburger','bagel with olives','bagel with raisins']
foods1_id = list(range(1, len(foods1)+1))
foods2 = ['bagel','pizza','salad with raisins','tuna and olives']
foods2_id = [chr(ord('a')+i) for i in range(len(foods2))]

# 生成句子嵌入
embeddings1 = model.encode(foods1, convert_to_tensor=True)
embeddings2 = model.encode(foods2, convert_to_tensor=True)

# 计算余弦相似度，找到每个食品的最佳匹配
matches = []
for i in range(len(foods1)):
    max_score_idx = util.cos_sim(embeddings1[i], embeddings2).argmax()
    matches.append({
        'foods1_id': foods1_id[i],
        'foods2_id': foods2_id[max_score_idx],
        'foods1': foods1[i],
        'foods2': foods2[max_score_idx],
        'similarity_score': util.cos_sim(embeddings1[i], embeddings2[max_score_idx]).item()
    })

final_matches = pd.DataFrame(matches)
print(final_matches)

运行后，"bagel with raisins"会匹配到"bagel"，因为二者语义更接近。

路径2：实体提取+分层匹配 #

类似R方案，用spaCy提取核心实体，先精确匹配再模糊匹配：

import spacy
import pandas as pd
from fuzzywuzzy import fuzz

# 加载spaCy英语模型（需先安装：pip install spacy && python -m spacy download en_core_web_sm）
nlp = spacy.load("en_core_web_sm")

# 提取核心食品实体函数
def extract_core_food(text):
    doc = nlp(text)
    # 取第一个名词/专有名词作为主食材
    for token in doc:
        if token.pos_ in ["NOUN", "PROPN"]:
            return token.text
    return text

# 原始数据
foods1 = ['bagel plain','bagel with raisins and olives', 'hamburger','bagel with olives','bagel with raisins']
foods1_id = list(range(1, len(foods1)+1))
foods2 = ['bagel','pizza','salad with raisins','tuna and olives']
foods2_id = [chr(ord('a')+i) for i in range(len(foods2))]

df1 = pd.DataFrame({'foods1_id': foods1_id, 'foods1': foods1})
df2 = pd.DataFrame({'foods2_id': foods2_id, 'foods2': foods2})

# 提取核心实体
df1['core_food'] = df1['foods1'].apply(extract_core_food)
df2['core_food'] = df2['foods2'].apply(extract_core_food)

# 精确匹配核心实体
exact_matches = pd.merge(df1, df2, on='core_food', how='inner')[['foods1_id', 'foods2_id', 'foods1', 'foods2']]

# 处理未匹配记录，结合实体相似度和模糊匹配
unmatched_df1 = df1[~df1['foods1_id'].isin(exact_matches['foods1_id'])]
unmatched_df2 = df2[~df2['foods2_id'].isin(exact_matches['foods2_id'])]

def get_best_match(row):
    # 计算候选的实体相似度和整体模糊相似度
    candidates = unmatched_df2.apply(lambda x: (
        x['foods2_id'], x['foods2'],
        fuzz.ratio(row['core_food'], x['core_food']),
        fuzz.WRatio(row['foods1'], x['foods2'])
    ), axis=1)
    # 优先实体相似度，再整体相似度排序
    candidates = candidates.sort_values(by=[2,3], ascending=False)
    best = candidates.iloc[0]
    return pd.Series([best[0], best[1]], index=['foods2_id', 'foods2'])

fuzzy_matches = unmatched_df1.apply(get_best_match, axis=1)
fuzzy_matches = pd.concat([unmatched_df1[['foods1_id', 'foods1']], fuzzy_matches], axis=1)

# 合并结果
final_matches = pd.concat([exact_matches, fuzzy_matches], ignore_index=True)
print(final_matches)

内容的提问来源于stack exchange，提问作者User981636