替换Fuzzywuzzy为Rapidfuzz后匹配无结果问题咨询
问题:替换Fuzzywuzzy为Rapidfuzz后匹配无结果?
我正在开发一套基于相似度匹配的系统,用于映射客户端数据与中央数据库的行数据,采用混合方法处理产品的厂商、型号、变体等特征——针对字符串类型字段,根据字段特性选择模糊逻辑或SBERT嵌入计算相似度。
原本模型运行效果尚可但速度偏慢,为提升性能,我将Fuzzywuzzy替换为更快的Rapidfuzz,仅修改了导入语句(从from fuzzywuzzy import fuzz改为from rapidfuzz import fuzz)。测试发现模糊分数计算结果和Fuzzywuzzy完全一致,但最终匹配无结果返回。
我尝试过将返回类型转换为float、调整匹配阈值等操作,均未解决问题;只有当阈值设为0时,匹配才恢复正常,但不确定这种方案是否合理。
相关代码如下:
def cached_fuzzy_score(a, b): return fuzz.token_set_ratio(a, b)/ 100
def get_candidates_vectorized(bank_make, central_df, threshold=60): # Use fuzzy matching on make names make_scores = central_df['make_name'].apply( lambda x: fuzz.token_set_ratio(bank_make, x) ) return central_df[make_scores > threshold].index.tolist() # Optimized scoring function def calculate_scores_batch(bank_rows, central_indices, central_df, model_vectorizer, segment_vectorizer, model_matrix, segment_matrix, segment_embeddings, identity_embeddings, central_variant_embeddings, central_identity_embeddings, weights): results = [] bank_models = [row['bank_model'] for row in bank_rows] model_sims = batch_semantic_similarity(bank_models, model_vectorizer, model_matrix) # Convert embeddings to torch tensors import torch bank_segment_embeddings = torch.stack(segment_embeddings) bank_identity_embeddings = torch.stack(identity_embeddings) sbert_segment_sims = util.pytorch_cos_sim(bank_segment_embeddings, central_variant_embeddings) sbert_identity_sims = util.pytorch_cos_sim(bank_identity_embeddings, central_identity_embeddings) for i, bank_row in enumerate(bank_rows): best_score = 0 best_match_idx = None for central_idx in central_indices[i]: central_row = central_df.iloc[central_idx] make_score = cached_fuzzy_score(bank_row['bank_make'], central_row['make_name']) model_score = model_sims[i, central_idx] segment_score = sbert_segment_sims[i][central_idx].item() fuel_score = cached_fuzzy_score( str(bank_row.get('extracted_fuel_type', '')), str(central_row['fuel_type']) ) transmission_score = cached_fuzzy_score( str(bank_row.get('transmission_type', '')), str(central_row['transmission']) ) displacement_score = cached_fuzzy_score( str(bank_row.get('extracted_displacement', '')), str(central_row['displacement_formatted']) ) identity_score = sbert_identity_sims[i][central_idx].item() # BS rating comparison bank_bs = bank_row.get('bs_rating') central_bs = central_row.get('bs_rating') if bank_bs is not None and central_bs is not None: try: bs_score = 1.0 - (abs(float(bank_bs) - float(central_bs)) / 3.0) bs_score = max(bs_score, 0.0) except: bs_score = cached_fuzzy_score(str(bank_bs), str(central_bs)) else: bs_score = 0.0 total_score = ( weights['make'] * make_score + weights['model'] * model_score + weights['segment'] * segment_score + weights['fuel'] * fuel_score + weights['transmission'] * transmission_score + weights['displacement'] * displacement_score + weights.get('bs', 0) * bs_score + weights.get('identity', 0.1) * identity_score ) if total_score > best_score: best_score = total_score best_match_idx = central_idx results.append((best_match_idx, best_score)) return results
内容的提问来源于stack exchange,提问作者Prabhjit Singh
相关产品推荐
相关产品推荐

