如何在大尺寸2D网格的z值中检测强度异常值？

场景说明 #

x和y的取值范围均为1到16（步长1），统计每对(xₙ,yₙ)的观测次数得到z值，形成16×16的网格；由于x与y的相关性约为0.5，预期部分区域的观测点会更密集，但设备错误导致低预期区域出现大量观测。现需针对9000×9000的大尺寸网格数据解决该异常检测问题。

示例代码 #

import pandas as pd
import random
import matplotlib.pyplot as plt
# 生成x、y、z数据，x和y类似矩阵坐标
x = [i for i in range(1, 17) for j in range(16)]
y = list(range(1, 17)) * 16

# 生成指定基数范围内的随机整数列表
def r_num(base_value: int, n_numbers: int):
    return [random.randint(base_value, base_value + 700) for i in range(n_numbers)]

def z_make():
    _z = ((r_num(2000, 16)) +
         (r_num(2000, 16)) +
         (r_num(2000, 2) + r_num(5000, 12) + r_num(2000, 2)) +
         (r_num(2000, 2) + r_num(5000, 12) + r_num(2000, 2)) +
         (r_num(2000, 2) + r_num(5000, 2) + r_num(7000, 8) + r_num(5000, 2) + r_num(2000, 2)) +
         (r_num(2000, 2) + r_num(5000, 2) + r_num(7000, 1) + r_num(9000, 6) + r_num(7000, 1) + r_num(5000, 2) + r_num(2000, 2)) +
         (r_num(2000, 2) + r_num(5000, 2) + r_num(7000, 1) + r_num(9000, 1) + r_num(10000, 4) + r_num(9000, 1) + r_num(7000, 1) + r_num(5000, 2) + r_num(2000, 2)) +
         (r_num(2000, 2) + r_num(5000, 2) + r_num(7000, 1) + r_num(9000, 1) + r_num(10000, 1) + r_num(16000, 2) + r_num(10000, 1) + r_num(9000, 1) + r_num(7000, 1) + r_num(5000, 2) + r_num(2000, 2)))
    return _z

z1 = z_make()
z2 = z_make()
z2.reverse()
z = z1 + z2

for i in ['x','y','z']:
    print('Len of i:', len(eval(i)))

df = pd.DataFrame({'x': x, 'y': y, 'z': z})

# 设置异常值和异常点附近的缺失值
df.loc[(df['x'] == 2) & (df['y'] == 6), 'z'] = 15875
df.loc[(df['x'] == 15) & (df['y'] == 2), 'z'] = 14999
df.loc[(df['x'] == 2) & (df['y'] == 7), 'z'] = None

plt.scatter(df.x, df.y, c=df.z, cmap='viridis', s=100, alpha=0.7)

# 添加颜色条
plt.colorbar(label='观测次数')

# 设置标签和标题
plt.xlabel('X坐标')
plt.ylabel('Y坐标')
plt.show()

可视化结果 #

从图中可见，异常点的z值远高于其邻域内的点。

已尝试方法及问题 #

曾尝试KDE、Contour和Local Outlier Factor (LOF)方法，但检测效果均不理想：

• KDE的检测效果受带宽参数影响极大，难以找到稳定适配大尺寸网格的参数；
• LOF等方法在大尺寸网格下性能和准确性不足。

现寻求能有效识别大尺寸网格中z值显著异于邻域点的异常检测方法。

内容的提问来源于stack exchange，提问作者Zoomman