如何在R语言中移除数据集里最大的5%距离数据？

为满足研究需求，需要移除数据集中最大的5%距离数据。通过代码quantile(TD2S1WP$distance,c(.95),na.rm = TRUE)计算出95%分位数为41.8，认为所有≥41.8的距离属于需移除的5%数据。随后尝试用代码TD2S1WP.2 <- TD2S1WP[!(TD2S1WP$distance>41.8),]进行筛选，但新数据集中仍存在41.8及更大的距离值。请问如何正确移除数据集中最大的5%距离数据？

数据集如下：

> dput(TD2S1WP)
structure(list(Region.Label = c("TD2S1", "TD2S1", "TD2S1", "TD2S1", 
"TD2S1", "TD2S1", "TD2S1", "TD2S1", "TD2S1", "TD2S1", "TD2S1", 
"TD2S1", "TD2S1", "TD2S1", "TD2S1", "TD2S1", "TD2S1", "TD2S1", 
"TD2S1", "TD2S1", "TD2S1", "TD2S1", "TD2S1", "TD2S1", "TD2S1", 
"TD2S1", "TD2S1", "TD2S1", "TD2S1", "TD2S1", "TD2S1", "TD2S1", 
"TD2S1", "TD2S1", "TD2S1", "TD2S1", "TD2S1", "TD2S1", "TD2S1", 
"TD2S1", "TD2S1", "TD2S1", "TD2S1", "TD2S1", "TD2S1", "TD2S1", 
"TD2S1", "TD2S1", "TD2S1", "TD2S1", "TD2S1"), Area = c(462, 462, 
462, 462, 462, 462, 462, 462, 462, 462, 462, 462, 462, 462, 462, 
462, 462, 462, 462, 462, 462, 462, 462, 462, 462, 462, 462, 462, 
462, 462, 462, 462, 462, 462, 462, 462, 462, 462, 462, 462, 462, 
462, 462, 462, 462, 462, 462, 462, 462, 462, 462), Sample.Label = 
c("11", 
"11", "6", "6", "6", "6", "6", "6", "6", "12", "12", "12", "12", 
"12", "3", "3", "3", "5", "5", "5", "5", "5", "5", "4", "9", 
"9", "9", "10", "10", "10", "10", "10", "10", "10", "10", "10", 
"8", "8", "8", "8", "8", "8", "8", "14", "14", "2", "13", "13", 
"1", "7", "15"), Effort = c(0.341, 0.341, 0.326, 0.326, 0.326, 
0.326, 0.326, 0.326, 0.326, 0.308, 0.308, 0.308, 0.308, 0.308, 
0.311, 0.311, 0.311, 0.315, 0.315, 0.315, 0.315, 0.315, 0.315, 
0.303, 0.34, 0.34, 0.34, 0.324, 0.324, 0.324, 0.324, 0.324, 0.324, 
0.324, 0.324, 0.324, 0.325, 0.325, 0.325, 0.325, 0.325, 0.325, 
0.325, 0.333, 0.333, 0.343, 0.318, 0.318, 0.322, 0.346, 0.312
), distance = c(0.6, 0.6, 21, 21, 7, 6.3, 16, 1.4, 1.4, 0.5, 
0.5, 1.4, 19, 4.8, 0.5, 0.5, 13, 41.8, 41.8, 41.8, 41.8, 41.8, 
41.8, 16.6, 8.5, 0.3, 8.6, 10.3, 10.3, 10.3, 10.3, 13.1, 13.1, 
13.1, 13.1, 13.1, 1.6, 1.6, 1, 2.3, 2.3, 14.1, 0.3, 7.7, 3.4, 
27, 16, 15.6, 21.3, NA, NA), Study.Area = c("TownDistrict2", 
"TownDistrict2", "TownDistrict2", "TownDistrict2", "TownDistrict2", 
"TownDistrict2", "TownDistrict2", "TownDistrict2", "TownDistrict2", 
"TownDistrict2", "TownDistrict2", "TownDistrict2", "TownDistrict2", 
"TownDistrict2", "TownDistrict2", "TownDistrict2", "TownDistrict2", 
"TownDistrict2", "TownDistrict2", "TownDistrict2", "TownDistrict2", 
"TownDistrict2", "TownDistrict2", "TownDistrict2", "TownDistrict2", 
"TownDistrict2", "TownDistrict2", "TownDistrict2", "TownDistrict2", 
"TownDistrict2", "TownDistrict2", "TownDistrict2", "TownDistrict2", 
"TownDistrict2", "TownDistrict2", "TownDistrict2", "TownDistrict2", 
"TownDistrict2", "TownDistrict2", "TownDistrict2", "TownDistrict2", 
"TownDistrict2", "TownDistrict2", "TownDistrict2", "TownDistrict2", 
"TownDistrict2", "TownDistrict2", "TownDistrict2", "TownDistrict2", 
"TownDistrict2", "TownDistrict2")), row.names = c(NA, -51L), class = 
c("tbl_df", 
"tbl", "data.frame"))

问题根源 #

你写的筛选条件!(TD2S1WP$distance>41.8)只会剔除大于41.8的数值，但等于41.8的所有记录都被保留了——而这些41.8正好属于你要移除的顶部5%。另外你的数据集去掉2个NA后有49条有效数据，5%约为2.5条，但数据里有6个41.8，这是因为quantile默认的计算逻辑会把重复的高值都归到95%分位点上，导致分位数对应了多个重复项。

两种可行解决方法 #

方法1：直接筛掉≥95%分位数的数值 #

把筛选逻辑改成保留小于41.8的数据，同时顺带排除NA：

# 先计算95%分位数
q95 <- quantile(TD2S1WP$distance, 0.95, na.rm = TRUE)
# 筛选距离小于q95且非NA的行
TD2S1WP.2 <- TD2S1WP[TD2S1WP$distance < q95 & !is.na(TD2S1WP$distance), ]

方法2：严格移除顶部5%样本 #

如果想不管分位数重复情况，直接删掉最大的5%样本，用排名或切片的方式更精准：

基础R实现

# 统计有效样本数（去掉NA）
valid_n <- sum(!is.na(TD2S1WP$distance))
# 计算要保留的行数：取95%的整数部分
keep_n <- floor(valid_n * 0.95)
# 按距离从小到大排序，取前keep_n行
TD2S1WP.2 <- TD2S1WP[order(TD2S1WP$distance), ][1:keep_n, ]

dplyr实现（更简洁）

如果习惯用tidyverse工具链：

library(dplyr)
TD2S1WP.2 <- TD2S1WP %>%
  filter(!is.na(distance)) %>%  # 先移除NA
  slice_head(prop = 0.95)       # 保留前95%的样本

验证结果 #

运行代码后，可通过max(TD2S1WP.2$distance)查看新数据集的最大距离，确认41.8已被移除。

内容的提问来源于stack exchange，提问作者Eonear Black