R语言提取数据框唯一字符串遇编码问题的解决问询

问题场景 #

有如下示例数据框：

dummy <- structure(list(location = c("RD WEST - ACOUSTIC RELEASE", "RD NORTH - ACOUSTIC RELEASE", 
"RD EAST - ACOUSTIC RELEASE", "R SOUTHWEST REEF", "RD NORTH - ACOUSTIC RELEASE", 
"RD EAST - ACOUSTIC RELEASE", "RD WEST - ACOUSTIC RELEASE"
)), row.names = c(NA, -7L), class = c("tbl_df", "tbl", "data.frame"
))

# A tibble: 7 × 1
  location                   
  <chr>                      
1 RD WEST - ACOUSTIC RELEASE 
2 RD NORTH - ACOUSTIC RELEASE
3 RD EAST - ACOUSTIC RELEASE 
4 R SOUTHWEST REEF           
5 RD NORTH - ACOUSTIC RELEASE
6 RD EAST - ACOUSTIC RELEASE 
7 RD WEST - ACOUSTIC RELEASE 

运行unique(dummy$location)本应返回4个唯一字符串，但实际返回全部7个值。

排查过程 #

1. 编码检查：通过Encoding(dummy$location)查看字符串编码，发现前4个字符串编码为"unknown"，后3个为"UTF-8"。
2. 尝试多种编码转换方法均无效，包括：

dummy$location <- iconv(dummy$location, from = "unknown", to = "UTF-8")

Encoding(dummy$location) <- "UTF-8"

dummy$location <- enc2utf8(dummy$location)

library(stringi)
dummy$location <- stri_encode(dummy$location, "", "UTF-8")

3. 字符底层检查：循环调用charToRaw()后发现，看似相同的字符串实际存在半角空格（十六进制20）与全角空格（十六进制c2 a0）的差异，这才是导致unique()无法识别重复值的根本原因，编码差异只是表象。

解决方案 #

使用fedmatch包的clean_strings()函数处理字符串后，成功提取到4个唯一值。此时即使所有字符串编码统一为"unknown"，也能正常使用unique()函数：

library(fedmatch)
dummy$location <- clean_strings(dummy$location)
unique(dummy$location)

内容的提问来源于stack exchange，提问作者FlyingDutch