如何按近似(非完全匹配)值合并两个DataFrame?
问题描述
我有两个DataFrame:
player_stats:
player minutes total_points assists 1 Erling Haaland 77 13 0 2 Kevin De Bruyne 90 6 1
season_gw1:
player position gw team 10449 Erling Håland 4 1 Manchester City 10453 Kevin De Bruyne 3 1 Manchester City
我想按player列合并这两个DataFrame,但第一位球员Haaland的拼写在两个DataFrame里不一致(一个是Haaland,一个是Håland)。我当前用的合并代码是:
season_gw1_stats = season_gw1.merge(player_stats, on = 'player')
合并后只得到了Kevin De Bruyne的数据:
player position gw team minutes total_points assists 10453 Kevin de Bruyne 3 1 Manchester City 90 6 1
请问如何实现按近似值合并DataFrame?(实际数据里还有其他拼写相似但不一致的名称需要处理)
解决方案
方法1:统一字符编码(处理特殊字符)
很多拼写差异源于特殊字符(如å、é这类),可以先将两个DataFrame的player列统一转成ASCII字符,消除这类差异:
import unicodedata def normalize_name(name): # 将特殊字符转成对应ASCII字符,比如å→a return unicodedata.normalize('NFKD', name).encode('ascii', 'ignore').decode('utf-8') # 对两个DataFrame的player列做归一化 player_stats['player_normalized'] = player_stats['player'].apply(normalize_name) season_gw1['player_normalized'] = season_gw1['player'].apply(normalize_name) # 用归一化后的列合并 season_gw1_stats = season_gw1.merge(player_stats, on='player_normalized').drop('player_normalized', axis=1)
这种方法适合处理因特殊字符、大小写导致的差异,比如Håland会被转成Haaland,和另一个DataFrame的名称完全匹配。
方法2:模糊匹配(用rapidfuzz)
如果存在更复杂的拼写差异(如缩写、少字母),可以用模糊匹配库计算字符串相似度,找到最匹配的条目。rapidfuzz是性能更优的模糊匹配工具:
先安装库:
pip install rapidfuzz
然后编写匹配逻辑:
from rapidfuzz import process, fuzz # 提取player_stats里的球员名称列表 player_names = player_stats['player'].tolist() # 给season_gw1的每个球员找到最匹配的player_stats里的名称,设置相似度阈值 def get_best_match(name, threshold=80): match, score, _ = process.extractOne(name, player_names, scorer=fuzz.ratio) return match if score >= threshold else None season_gw1['matched_player'] = season_gw1['player'].apply(get_best_match) # 用匹配后的列合并 season_gw1_stats = season_gw1.merge(player_stats, left_on='matched_player', right_on='player').drop('matched_player', axis=1)
- 可调整
threshold(默认80)控制匹配严格程度,分数越高匹配越严格。 fuzz.ratio计算整体相似度,也可使用fuzz.partial_ratio处理前缀/后缀差异的场景。
方法3:结合两种方法(先归一化再模糊匹配)
如果同时存在特殊字符和拼写差异,先做字符归一化再进行模糊匹配,效果会更稳定:
# 先做字符归一化 player_stats['player_normalized'] = player_stats['player'].apply(normalize_name) season_gw1['player_normalized'] = season_gw1['player'].apply(normalize_name) # 用归一化后的名称做模糊匹配 normalized_names = player_stats['player_normalized'].tolist() def get_best_normalized_match(name, threshold=80): match, score, _ = process.extractOne(name, normalized_names, scorer=fuzz.ratio) return match if score >= threshold else None season_gw1['matched_normalized'] = season_gw1['player_normalized'].apply(get_best_normalized_match) # 合并时关联回原数据 season_gw1_stats = season_gw1.merge( player_stats, left_on='matched_normalized', right_on='player_normalized' ).drop(['player_normalized_x', 'player_normalized_y', 'matched_normalized'], axis=1)
内容的提问来源于stack exchange,提问作者grungrun
相关产品推荐
相关产品推荐

