使用readr::read_csv解析带转义字符串的CSV文件遇问题求助
关于readr::read_csv解析带引号的特殊格式CSV文件的问题
我有一个CSV文件,其中数字以字符串形式转义,且小数点为逗号。之前用readr::read_csv解析过这类文件,但现在遇到问题,引号包裹的字符串未被正确识别。
测试场景1:千分位为逗号且被引号包裹
执行代码及结果:
# This will go okay, except for an unecessary warning read_csv( 'A, B, C, D, E $1, a, a, 1.2%, "1,100,200" $2, b, b, 2.1%, " 140,000" $3, c, c, 13.0%, "2,005,000"', col_types = "nccnn" )
输出:
# A tibble: 3 × 5 A B C D E <dbl> <chr> <chr> <dbl> <dbl> 1 1 a a 1.2 1100200 2 2 b b 2.1 140000 3 3 c c 13 2005000 Warning message: One or more parsing issues, see `problems()` for details
说明:虽能正确解析最后一列,但会抛出不必要的列数错误警告。
测试场景2:小数点为逗号、千分位为句点
执行代码及结果:
# This will go all wrong, because the locale is not respected if ',' is a separator inside a string read_csv( 'A, B, C, D, E $1, a, a, " 1,2%", "1.100.200" $2, b, b, " 2,1%", " 140.000" $3, c, c, "13,0%", "2.005.000"', locale=locale(decimal_mark = ",", grouping_mark = "."), col_types = "nccnn")
输出:
# A tibble: 3 × 5 A B C D E <dbl> <chr> <chr> <dbl> <dbl> 1 1 a a 1 2 2 2 b b 2 1 3 3 c c 13 0 Warning message: One or more parsing issues, see `problems()` for details
说明:引号内的逗号被当作字段分隔符,百分比内容被拆分为两列,最后一列直接丢失。
疑问
请问引号不应包裹逗号使其不被识别为字段分隔符吗?若无法实现,有无无需修改输入文件的解决办法?
解决办法
CSV规范中,引号包裹内容里的逗号应该被当作内容而非分隔符,但readr在结合locale设置时会出现解析冲突——当把decimal_mark设为逗号时,解析器会优先将逗号识别为小数点,导致引号内的逗号被错误拆分。
针对两个场景的解决方案:
场景1:千分位为逗号(无小数点冲突)
警告源于read_csv预解析时把引号内的逗号当成潜在分隔符,导致列数预判错误,但实际解析能正确识别引号内容。可通过指定guess_max参数消除警告:
read_csv( 'A, B, C, D, E $1, a, a, 1.2%, "1,100,200" $2, b, b, 2.1%, " 140,000" $3, c, c, 13.0%, "2,005,000"', col_types = "nccnn", guess_max = 3 )
让解析器完整读取所有行后再判断列数,即可避免列数预判警告。
场景2:小数点为逗号、千分位为句点(引号内逗号被误拆分)
核心问题是decimal_mark = ","与CSV分隔符逗号的冲突,解决思路是先读取所有内容为字符列,再手动转换格式:
- 先读取所有列为字符,避免解析错误:
raw_data <- read_csv( 'A, B, C, D, E $1, a, a, " 1,2%", "1.100.200" $2, b, b, " 2,1%", " 140.000" $3, c, c, "13,0%", "2.005.000"', col_types = "ccccc" )
- 结合指定locale用
readr解析函数处理每一列:
library(dplyr) clean_data <- raw_data %>% mutate( A = parse_number(A), D = parse_number(D, locale = locale(decimal_mark = ",")), E = parse_number(E, locale = locale(grouping_mark = ".")) )
处理后得到正确结果:
# A tibble: 3 × 5 A B C D E <dbl> <chr> <chr> <dbl> <dbl> 1 1 a a 1.2 1100200 2 2 b b 2.1 140000 3 3 c c 13 2005000
另外,也可以用data.table::fread替代,它对带引号的特殊格式CSV解析更稳健,无需额外设置就能正确识别引号内的逗号:
library(data.table) fread( 'A, B, C, D, E $1, a, a, " 1,2%", "1.100.200" $2, b, b, " 2,1%", " 140.000" $3, c, c, "13,0%", "2.005.000"', dec = ",", sep = "," )
输出会自动处理引号内的逗号,并正确解析小数点和千分位。
内容的提问来源于stack exchange,提问作者Thomas Mailund
相关产品推荐
相关产品推荐

