如何在R中清理地址自连接匹配结果中的重复反向冗余行
问题描述
我执行了自连接操作来对地址数据做相互比对,所用代码如下:
library(tidyverse) library(lubridate) library(stringr) library(stringdist) library(fuzzyjoin) doTheJoin <- function (threshold) { joined <- trimData(d_client_squashed) %>% stringdist_left_join( trimData(d_client_squashed), by = c(address_full="address_full"), distance_col = "distance", max_dist = threshold, method = "jw" ) }
d_client_squashed的结构如下,存储的均为字符串类型值:
| Client_Reference | adress_full |
|---|---|
| C01 | Client1 Name, Street, Zipcode, Town |
| C02 | Client2 Name, Street2, Zipcode2, Town2 |
| ... | ... |
随后我执行了如下操作:
sensible_matches <- doTheJoin(0.2) View(sensible_matches %>% filter(Client_Reference.x != Client_Reference.y))
得到的输出结果如下:
| Client_Reference.x | address_full.x | Client_Reference.y | address_full.y | Distance |
|---|---|---|---|---|
| C01 | Client1 Name, Street, Zipcode, Town | C02 | Client2 Name, Street2, Zipcode2, Town2 | 0.05486 |
| C02 | Client2 Name, Street2, Zipcode2, Town2 | C01 | Client1 Name, Street, Zipcode, Town | 0.05486 |
| ... | ... | ... | ... | ... |
目前连接操作的输出存在重复问题:每组匹配都会生成两条客户信息颠倒的记录,且距离值不唯一,无法用作去重依据。请问我该如何对该数据框取子集,以去除这类重复行?
解决方案
这里提供两种可直接落地的实现方式,按需选择即可:
方案1:匹配完成后直接去重
如果已经跑完匹配不想重新计算,可以新增固定顺序的配对标识直接去重:
sensible_matches %>% # 先过滤掉自身匹配的条目 filter(Client_Reference.x != Client_Reference.y) %>% # 生成配对唯一标识:将两个客户参考号按字典序排序后拼接,保证正向反向配对生成的标识完全一致 mutate(match_pair = pmap_chr(list(Client_Reference.x, Client_Reference.y), ~paste(sort(c(..1, ..2)), collapse = "_"))) %>% # 按配对标识去重,每组仅保留1条记录 distinct(match_pair, .keep_all = TRUE) %>% # 可选:删除辅助列 select(-match_pair)
方案2:自连接时从源头避免生成重复条目
数据量较大时更推荐这个方案,可以减少一半的匹配计算量,直接修改doTheJoin函数即可:
doTheJoin <- function (threshold) { trimmed_data <- trimData(d_client_squashed) %>% # 新增行号作为辅助判断字段 mutate(row_id = row_number()) joined <- trimmed_data %>% stringdist_left_join( trimmed_data, by = c(address_full="address_full"), distance_col = "distance", max_dist = threshold, method = "jw" ) %>% # 仅保留x行号小于y行号的匹配结果,天然避免双向重复和自身匹配 filter(row_id.x < row_id.y) %>% # 删掉不需要的行号辅助列 select(-starts_with("row_id")) }
内容的提问来源于stack exchange,提问作者Johannes Stephan
相关产品推荐
相关产品推荐

