R中全字符列类型的data.table如何删除NaN与Inf值
data.table快速删除含NaN/Inf行的解决方案
我有如下data.table:
data = structure(list(date = c("2021-11-24", "2021-11-24", "2021-11-26", "2021-11-24", "2021-11-26", "2021-11-24", "2021-11-24", "2021-11-26", "2021-11-26", "2021-11-26", "2021-11-26"), open = c("NaN", "NaN", "0.43", "0.17", "0.19", "0.15", "NaN", "NaN", "NaN", "NaN", "NaN" ), high = c("NaN", "NaN", "0.43", "0.17", "0.19", "0.15", "NaN", "NaN", "NaN", "NaN", "NaN"), low = c("NaN", "NaN", "0.43", "0.17", "0.19", "0.15", "NaN", "NaN", "NaN", "NaN", "NaN"), close = c("NaN", "NaN", "0.43", "0.17", "0.19", "0.15", "NaN", "NaN", "NaN", "NaN", "NaN"), volume = c(0L, 0L, 2L, 10L, 75L, 1L, 0L, 0L, 0L, 0L, 0L)), row.names = c(NA, -11L), class = c("data.table", "data.frame" ))
我需要删除该data.table中所有NaN和Inf值,原始数据预览如下:
date open high low close volume 1: 2021-11-24 NaN NaN NaN NaN 0 2: 2021-11-24 NaN NaN NaN NaN 0 3: 2021-11-26 0.43 0.43 0.43 0.43 2 4: 2021-11-24 0.17 0.17 0.17 0.17 10 5: 2021-11-26 0.19 0.19 0.19 0.19 75 6: 2021-11-24 0.15 0.15 0.15 0.15 1 7: 2021-11-24 NaN NaN NaN NaN 0 8: 2021-11-26 NaN NaN NaN NaN 0 9: 2021-11-26 NaN NaN NaN NaN 0 10: 2021-11-26 NaN NaN NaN NaN 0 11: 2021-11-26 NaN NaN NaN NaN 0
受NaN值影响,open、high、low、close列均为字符类型,请问有没有可直接在data.table中快速删除NaN的方法?
各方案性能测试结果
以下是不同实现方案的性能对比测试代码:
p_load(dtplyr, dplyr) microbenchmark::microbenchmark( user438383 = data[!unique(which(data == "NaN" | data == "Inf", arr.ind=T)[,1])], langtang = na.omit(cbind(data[, .(date,volume)], data[, lapply(.SD, as.numeric), .SDcols = 2:5])), akrun = {data <- type.convert(data, as.is = TRUE); data[data[, Reduce(`&`, lapply(.SD, function(x) !is.nan(x) & is.finite(x))), .SDcols = -1]]}, paul = {data <- type.convert(data, as.is = TRUE); data[data[,is.finite(rowSums(.SD)), .SDcols=-1]]}, Macosso = {data$Row <- row.names(data); rm_rw <- data[apply(data, 1, function(X) any(X== "NaN"|X== "Inf")),] %>% dplyr::pull(Row); data[!row.names(data) %in% rm_rw ,] %>% dplyr::select(-Row)} )
测试输出结果:
Unit: microseconds expr min lq mean median uq max neval cld user438383 893.843 931.243 976.4554 974.011 1005.673 1093.929 100 a langtang 2694.987 2779.411 2904.5124 2877.927 3003.832 3420.539 100 c akrun 1664.476 1694.780 2253.8962 1731.392 1838.755 26035.268 100 b paul 1663.552 1718.956 1792.2313 1770.511 1843.051 2151.975 100 b Macosso 5899.961 6140.244 6429.9634 6368.072 6604.615 8180.782 100 d
从测试结果可以看出,user438383提供的方案性能最优,无需提前转换列数据类型,直接匹配字符串形式的NaN和Inf定位待删除行,执行速度是其他方案的2-6倍。
内容的提问来源于stack exchange,提问作者Saurabh
相关产品推荐
相关产品推荐

