在R中基于模糊条件分块去重：解决compare.dedup返回NA问题

数据集定义 #

address = c( "44 Ocean Road Atlanta Georgia", "882 4N Road River NY, NY 12345", "882 - River Road NY, ZIP 12345", "123 Fake Road Boston Drive Boston", "123 Fake - Rd Boston 56789", "3665 Apt 5 Moon Crs", "3665 Unit Moon Crescent", "NO ADDRESS PROVIDED", "31 Silver Way Road", "1800 Orleans St, Baltimore, MD 21287, United States", 
"1799 Orlans Street, Maryland , USA")
            
name = c("Pancake House of America" ,"ABC Center Building", "Cent. Bldg ABC", "BD Home 25 New", "Boarding Direct 25", "Pine Recreational Center", "Pine Rec. cntR", "Boston Swimming Complex", "boston gym center", "mas hospital" , "Massachusetts Hospital" )

blocking_var = c(1, 1,1,1, 1, 2,2,2,2,3,3)
            
my_data = data.frame(address, name, blocking_var)

数据预览 #

> my_data
                                               address                     name blocking_var
1                        44 Ocean Road Atlanta Georgia Pancake House of America            1
2                       882 4N Road River NY, NY 12345      ABC Center Building            1
3                       882 - River Road NY, ZIP 12345           Cent. Bldg ABC            1
4                    123 Fake Road Boston Drive Boston           BD Home 25 New            1
5                           123 Fake - Rd Boston 56789       Boarding Direct 25            1
6                                  3665 Apt 5 Moon Crs Pine Recreational Center            2
7                              3665 Unit Moon Crescent           Pine Rec. cntR            2
8                                  NO ADDRESS PROVIDED  Boston Swimming Complex            2
9                                   31 Silver Way Road        boston gym center            2
10 1800 Orleans St, Baltimore, MD 21287, United States             mas hospital            3
11                  1799 Orlans Street, Maryland , USA   Massachusetts Hospital            3

需求与问题 #

要按blocking_var分组，在每个组内去除模糊重复记录，仅保留唯一项。使用RecordLinkage包的compare.dedup()函数时，结果全为NA，无法继续后续去重操作。

尝试的代码 #

library(RecordLinkage)
pairs=compare.dedup(my_data, blockfld=3)

期望输出 #

address                     name blocking_var
1                        44 Ocean Road Atlanta Georgia Pancake House of America            1
2                       882 4N Road River NY, NY 12345      ABC Center Building            1
4                    123 Fake Road Boston Drive Boston           BD Home 25 New            1
6                                  3665 Apt 5 Moon Crs Pine Recreational Center            2
9                                   31 Silver Way Road        boston gym center            2
10 1800 Orleans St, Baltimore, MD 21287, United States             mas hospital            3

解决方案 #

问题根源 #

compare.dedup()默认会将所有字段按数值类型处理，而你的address和name是字符串，且未指定模糊匹配规则，导致计算时返回NA。必须明确指定字符串字段的比较方法。

分步解决代码 #

1. 加载包并配置模糊比较规则
   指定对address和name字段使用Jaro-Winkler模糊匹配算法，同时按blocking_var分组：

   library(RecordLinkage)
   # 定义比较规则：前两列是字符串用模糊匹配，第三列是分组字段不参与比较
   pairs <- compare.dedup(my_data, 
                          blockfld = "blocking_var",
                          strcmp = c(TRUE, TRUE, FALSE),
                          strcmpfun = jarowinkler)
   

2. 计算匹配权重并分类重复项
   用epiWeights()计算匹配权重，再通过阈值判断是否为重复记录：

   weights <- epiWeights(pairs)
   # 设定阈值（可根据数据调整，这里用0.8作为重复判定标准）
   classified <- classify(weights, threshold = 0.8)
   

3. 提取去重后的记录
   筛选出非重复的记录，得到最终结果：

   # 获取唯一记录的索引
   unique_idx <- which(!duplicated(classified$prediction))
   deduplicated_data <- my_data[unique_idx, ]
   # 重置行名
   rownames(deduplicated_data) <- NULL
   

分组精准处理（可选） #

如果需要更严格地按每个分组单独处理，避免跨组干扰，可使用分组循环：

deduplicated_data <- do.call(rbind, lapply(split(my_data, my_data$blocking_var), function(block) {
  block_pairs <- compare.dedup(block, strcmp = c(TRUE, TRUE, FALSE), strcmpfun = jarowinkler)
  block_weights <- epiWeights(block_pairs)
  block_class <- classify(block_weights, threshold = 0.8)
  block[!duplicated(block_class$prediction), ]
}))
rownames(deduplicated_data) <- NULL

关键说明 #

• strcmp参数：标记哪些字段是字符串类型，需要做模糊比较。
• strcmpfun：选择合适的模糊匹配算法，Jaro-Winkler适合地址、名称这类短字符串的匹配。
• 阈值调整：如果重复项识别不准确，可微调threshold值（比如0.7-0.9之间）。

内容的提问来源于stack exchange，提问作者stats_noob