如何在Python中实现Fruit列自模糊匹配并生成Cluster聚类列？

解决方案 #

可以通过模糊字符串匹配结合分组/聚类的方式实现，下面提供两种实用方法：

方法1：基于模糊匹配的手动分组（简单直观）

使用rapidfuzz库计算字符串相似度，将相似度高于阈值的水果归为同一簇：

import pandas as pd
from rapidfuzz import process, fuzz
from collections import defaultdict

# 示例数据（替换为你的实际数据即可）
data = {
    'Fruit': ['apple', 'apples', 'banana', 'bananas', 'orange', 'oranges', 'grape', 'grapes', 'strawberry', 'strawberries']
}
df = pd.DataFrame(data)

def create_fruit_clusters(fruit_list, similarity_threshold=80):
    clusters = defaultdict(list)
    processed_fruits = set()
    
    for fruit in fruit_list:
        if fruit in processed_fruits:
            continue
        # 提取所有相似度达标的匹配项
        matched_items = process.extract(fruit, fruit_list, scorer=fuzz.ratio, limit=None)
        cluster_members = [item[0] for item in matched_items if item[1] >= similarity_threshold]
        clusters[fruit] = cluster_members
        processed_fruits.update(cluster_members)
    
    # 生成水果到簇名的映射关系
    fruit_to_cluster = {member: cluster_name for cluster_name, members in clusters.items() for member in members}
    return [fruit_to_cluster[fruit] for fruit in fruit_list]

# 生成Cluster列
df['Cluster'] = create_fruit_clusters(df['Fruit'].tolist())
print(df)

输出结果：

Fruit   Cluster
0       apple     apple
1      apples     apple
2      banana    banana
3     bananas    banana
4      orange    orange
5     oranges    orange
6       grape     grape
7      grapes     grape
8  strawberry  strawberry
9 strawberries  strawberry

方法2：基于DBSCAN的自动聚类（适合复杂数据集）

通过计算字符串间的编辑距离，结合DBSCAN聚类算法自动完成分组：

import pandas as pd
import numpy as np
from sklearn.cluster import DBSCAN
from sklearn.preprocessing import StandardScaler
from rapidfuzz.distance import Levenshtein

# 示例数据
data = {
    'Fruit': ['apple', 'apples', 'banana', 'bananas', 'orange', 'oranges', 'grape', 'grapes', 'strawberry', 'strawberries']
}
df = pd.DataFrame(data)

def cluster_fruits_dbscan(fruit_list):
    # 构建两两编辑距离矩阵
    num_fruits = len(fruit_list)
    distance_matrix = np.zeros((num_fruits, num_fruits))
    for i in range(num_fruits):
        for j in range(num_fruits):
            distance_matrix[i][j] = Levenshtein.distance(fruit_list[i], fruit_list[j])
    
    # 标准化距离数据
    scaler = StandardScaler()
    scaled_distances = scaler.fit_transform(distance_matrix)
    
    # DBSCAN聚类（eps为距离阈值，min_samples为簇的最小元素数量）
    dbscan = DBSCAN(eps=0.5, min_samples=2, metric='precomputed')
    cluster_labels = dbscan.fit_predict(scaled_distances)
    
    # 为每个簇分配代表名称
    cluster_name_map = {}
    for idx, label in enumerate(cluster_labels):
        if label not in cluster_name_map:
            cluster_name_map[label] = fruit_list[idx]
    
    return [cluster_name_map[label] for label in cluster_labels]

# 生成Cluster列
df['Cluster'] = cluster_fruits_dbscan(df['Fruit'].tolist())
print(df)

注意事项 #

1. 依赖安装：执行pip install rapidfuzz pandas scikit-learn安装所需库
2. 阈值调整：两种方法中的阈值（similarity_threshold或eps）需根据实际数据调整，确保相似水果能被正确分组
3. 性能优化：大规模数据集推荐使用rapidfuzz，它比旧的fuzzywuzzy速度快数倍

内容的提问来源于stack exchange，提问作者learn2learn