如何通过KQL查询基于错误内容去除重复日志条目

问题描述 #

当前使用KQL筛选容器错误日志时，因日志包含不同时间戳及额外字符，即使错误核心内容一致，distinct操作也无法实现有效去重，原查询语句如下：

let ContainerIdList = KubePodInventory
| where ContainerName contains "boss-logger"
| where Namespace has "devns"
| where ClusterId =~ '/subscriptions/abc1234-bcbd-3124-bc3a-524132fb1786645/resourcegroups/rg-aks-dev/providers/Microsoft.ContainerService/managedClusters/aksdevcluster'
| distinct ContainerID;
ContainerLog 
| where ContainerID in (ContainerIdList)
| where LogEntry matches regex "ERROR" 
| project LogEntry
| distinct LogEntry

解决方案 #

核心思路是提取ERROR标记后的核心错误内容，以此作为去重依据，以下是两种可行实现方式：

方法1：用extract()提取错误核心内容 #

通过正则匹配精准提取ERROR之后的核心错误文本，再基于该文本分组去重：

let ContainerIdList = KubePodInventory
| where ContainerName contains "boss-logger"
| where Namespace has "devns"
| where ClusterId =~ '/subscriptions/abc1234-bcbd-3124-bc3a-524132fb1786645/resourcegroups/rg-aks-dev/providers/Microsoft.ContainerService/managedClusters/aksdevcluster'
| distinct ContainerID;
ContainerLog 
| where ContainerID in (ContainerIdList)
| where LogEntry matches regex "ERROR"
// 提取ERROR后的核心内容，可根据实际日志格式调整正则（比如ERROR后无空格则去掉\s+）
| extend ErrorCore = extract(@"ERROR\s+(.*)", 1, LogEntry)
// 按核心错误内容分组，保留任意一条原始日志
| summarize any(LogEntry) by ErrorCore
| project LogEntry

方法2：用replace()去除无关内容 #

若时间戳、前缀等无关内容有固定格式，可直接替换为空后再去重：

let ContainerIdList = KubePodInventory
| where ContainerName contains "boss-logger"
| where Namespace has "devns"
| where ClusterId =~ '/subscriptions/abc1234-bcbd-3124-bc3a-524132fb1786645/resourcegroups/rg-aks-dev/providers/Microsoft.ContainerService/managedClusters/aksdevcluster'
| distinct ContainerID;
ContainerLog 
| where ContainerID in (ContainerIdList)
| where LogEntry matches regex "ERROR"
// 替换固定格式的时间戳前缀为空，可根据实际日志调整正则
| extend CleanedLog = replace(@"^\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}\s+", "", LogEntry)
// 基于清理后的内容去重，保留原始日志
| summarize any(LogEntry) by CleanedLog
| project LogEntry

注意事项 #

• 正则表达式需匹配实际日志格式，比如错误内容含换行时，可改用[\s\S]*匹配任意字符；
• 若ERROR标记前后有特殊符号，需同步调整正则的匹配规则。

内容的提问来源于stack exchange，提问作者ramesh reddy