R语言实现data.table数据y轴平滑后外推填充缺失值

需求说明 #

在如下data.table数据集中，需要完成两个处理步骤：

• 首先对所有非NA的y值做平滑处理，修正偏离整体曲线趋势的异常点，使其贴合曲线整体走向。例如P1分组中，x=77.5对应的y值0.850明显偏离趋势，按照前序点递减的规律，该点y值应当小于前序点（x=77）的y值；
• 平滑处理完成后，通过外推法补全所有y列的NA缺失值。

测试数据 #

test_dt = structure(list(group = c("P1", "P1", "P1", "P1", "P1", "P1", 
"P1", "P1", "P1", "P1", "P2", "P2", "P2", "P2", "P2", "P2", "P2", 
"P2", "P2", "P2", "P2", "P2", "P2", "P2", "P2", "P2", "P2", "P2", 
"P2"), x = c(74, 75, 76, 77, 77.5, 78, 79, 80, 81, 82, 70, 81, 
82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 98, 
105), y = c(1.05, 0.575, 0.275, 0.125, 0.85, NA, NA, NA, NA, 
NA, 14.2, 3.7, 2.875, 2.175, 1.575, 1.1, 0.7, 0.475, 0.3, 0.65, 
0.125, 0.925, NA, NA, NA, NA, NA, NA, NA)), row.names = c(NA, 
-29L), class = c("data.table", "data.frame"))

数据集预览：

group     x      y
 1:    P1  74.0  1.050
 2:    P1  75.0  0.575
 3:    P1  76.0  0.275
 4:    P1  77.0  0.125
 5:    P1  77.5  0.850
 6:    P1  78.0     NA
 7:    P1  79.0     NA
 8:    P1  80.0     NA
 9:    P1  81.0     NA
10:    P1  82.0     NA
11:    P2  70.0 14.200
12:    P2  81.0  3.700
13:    P2  82.0  2.875
14:    P2  83.0  2.175
15:    P2  84.0  1.575
16:    P2  85.0  1.100
17:    P2  86.0  0.700
18:    P2  87.0  0.475
19:    P2  88.0  0.300
20:    P2  89.0  0.650
21:    P2  90.0  0.125
22:    P2  91.0  0.925
23:    P2  92.0     NA
24:    P2  93.0     NA
25:    P2  94.0     NA
26:    P2  95.0     NA
27:    P2  96.0     NA
28:    P2  98.0     NA
29:    P2 105.0     NA

绘图查看原始数据分布：

library(pacman)
p_load(ggplot2)
ggplot(test_dt, aes(x, y, group = group)) + geom_point() + geom_line()

现有尝试的问题 #

目前尝试使用lowess函数做平滑，先删除了所有NA值再按分组计算平滑值，但当前参数设置下完全没有修正偏离曲线的异常点，代码和运行结果如下：

test_dt = na.omit(test_dt)
test_dt[, y_1 := round(lowess(x = x, y = y, f = 0.1)$y, 3), by = c("group")]

运行输出：

group    x      y    y_1
 1:    P1 74.0  1.050  1.050
 2:    P1 75.0  0.575  0.575
 3:    P1 76.0  0.275  0.275
 4:    P1 77.0  0.125  0.125
 5:    P1 77.5  0.850  0.850
 6:    P2 70.0 14.200 14.200
 7:    P2 81.0  3.700  3.700
 8:    P2 82.0  2.875  2.875
 9:    P2 83.0  2.175  2.175
10:    P2 84.0  1.575  1.575
11:    P2 85.0  1.100  1.100
12:    P2 86.0  0.700  0.700
13:    P2 87.0  0.475  0.475
14:    P2 88.0  0.300  0.300
15:    P2 89.0  0.650  0.650
16:    P2 90.0  0.125  0.125
17:    P2 91.0  0.925  0.925

内容的提问来源于stack exchange，提问作者Saurabh