使用AWK统计日志IP出现次数遇异常，求代码问题排查

问题背景 #

我想用AWK从日志文件里解析100.64段的IP地址，统计每个IP的出现次数，期望每行输出唯一IP和对应的次数：

IP1 occurences
IP2 ocurrences

日志格式 #

日志行格式如下：

2023-03-30 07:14:32.494 INFO  [asda/1737.140291506677312] cmd <IP> has been allocated

实际示例日志：

2023-03-30 10:39:31.214 INFO  [kea-dhcp4.leases/1737.140291531855424] DHCP4_LEASE_ALLOC [hwtype=1<REDACTED>: lease 100.64.147.36 has been allocated for 43200 seconds
2023-03-30 10:39:31.598 INFO  [kea-dhcp4.leases/1737.140291506677312] DHCP4_LEASE_ALLOC [hwtype=1<REDACTED>: lease 100.64.146.13 has been allocated for 43200 seconds
2023-03-30 10:39:31.745 INFO  [kea-dhcp4.leases/1737.140291523462720] DHCP4_LEASE_ALLOC [hwtype=1<REDACTED>: lease 100.64.146.4 has been allocated for 43200 seconds
2023-03-30 10:39:32.396 INFO  [kea-dhcp4.leases/1737.140291515070016] DHCP4_LEASE_ALLOC [hwtype=1<REDACTED>: lease 100.64.147.17 has been allocated for 43200 seconds
2023-03-30 10:39:32.466 INFO  [kea-dhcp4.leases/1737.140291531855424] DHCP4_LEASE_ALLOC [hwtype=1<REDACTED>: lease 100.64.144.122 has been allocated for 43200 seconds
2023-03-30 10:39:33.079 INFO  [kea-dhcp4.leases/1737.140291506677312] DHCP4_LEASE_ALLOC [hwtype=1<REDACTED>: lease 100.64.144.161 has been allocated for 43200 seconds
2023-03-30 10:39:33.220 INFO  [kea-dhcp4.leases/1737.140291523462720] DHCP4_LEASE_ALLOC [hwtype=1<REDACTED>: lease 100.64.144.77 has been allocated for 43200 seconds
2023-03-30 10:39:33.407 INFO  [kea-dhcp4.leases/1737.140291515070016] DHCP4_LEASE_ALLOC [hwtype=1<REDACTED>: lease 100.64.147.66 has been allocated for 43200 seconds
2023-03-30 10:39:33.427 INFO  [kea-dhcp4.leases/1737.140291531855424] DHCP4_LEASE_ALLOC [hwtype=1<REDACTED>: lease 100.64.149.201 has been allocated for 43200 seconds
2023-03-30 10:39:33.839 INFO  [kea-dhcp4.leases/1737.140291506677312] DHCP4_LEASE_ALLOC [hwtype=1<REDACTED>: lease 100.64.146.45 has been allocated for 43200 seconds
2023-03-30 10:39:34.530 INFO  [kea-dhcp4.leases/1737.140291523462720] DHCP4_LEASE_ALLOC [hwtype=1<REDACTED>: lease 100.64.144.47 has been allocated for 43200 seconds
2023-03-30 10:39:35.098 INFO  [kea-dhcp4.leases/1737.140291515070016] DHCP4_LEASE_ALLOC [hwtype=1<REDACTED>: lease 100.64.144.30 has been allocated for 43200 seconds
2023-03-30 10:39:35.249 INFO  [kea-dhcp4.leases/1737.140291531855424] DHCP4_LEASE_ALLOC [hwtype=1<REDACTED>: lease 100.64.147.54 has been allocated for 43200 seconds
2023-03-30 10:39:36.081 INFO  [kea-dhcp4.leases/1737.140291506677312] DHCP4_LEASE_ALLOC [hwtype=1<REDACTED>: lease 100.64.144.77 has been allocated for 43200 seconds

错误的AWK代码 #

我用了这段代码：

awk '/100.64./{for(i=1;i<=NF;++i)if($i~/100.64./){a[$i]++} {for (x in a) print x,a[x]}}'

错误结果 #

得到的输出是重复IP、计数混乱的列表，部分内容如下：

100.64.147.36 1
100.64.147.36 1
100.64.146.13 1
100.64.147.36 1
100.64.146.13 1
100.64.146.4 1
100.64.147.36 1
...
100.64.147.36 1
100.64.144.30 1
100.64.144.161 1
100.64.147.66 1
100.64.146.13 1
100.64.149.201 1
100.64.146.4 1
100.64.144.47 1
100.64.146.45 1
100.64.144.122 1
100.64.144.77 2
100.64.147.17 1

期望结果 #

希望得到每个唯一IP和对应次数，按IP排序：

100.64.144.122 1
100.64.144.161 1
100.64.144.30 1
100.64.144.47 1
100.64.144.77 2
100.64.146.13 1
100.64.146.4 1
100.64.146.45 1
100.64.147.17 1
100.64.147.36 1
100.64.147.54 1
100.64.147.66 1
100.64.149.201 1

问题分析 #

你的AWK代码核心问题是在每一行匹配到IP后就立刻遍历输出整个统计数组，而不是等所有日志行处理完再输出最终结果。这会导致每处理一行，就把当前已经统计到的所有IP都打印一遍，所以出现大量重复的IP行，计数也会随着处理逐步更新，看起来混乱。

另外还有两个小问题：

1. 正则里的.是通配符，应该转义成\.，否则会匹配任意字符，比如100x64也会被误匹配
2. 遍历数组for(x in a)的输出顺序是随机的，无法保证IP排序

修正后的代码 #

基础版本（统计并输出，不排序） #

awk '/100\.64\./{for(i=1;i<=NF;++i)if($i~/100\.64\./){a[$i]++}} END{for(x in a) print x,a[x]}' logfile

• 把打印数组的for循环放到END块里，这样只会在所有行处理完成后，输出一次最终的统计结果
• 转义正则中的.，避免误匹配

排序版本（匹配期望结果的IP排序） #

如果需要按IP地址升序排列输出，可以用管道结合sort：

awk '/100\.64\./{for(i=1;i<=NF;++i)if($i~/100\.64\./){a[$i]++}} END{for(x in a) print x,a[x]}' logfile | sort -t. -k1,1n -k2,2n -k3,3n -k4,4n

• sort -t. -k1,1n -k2,2n -k3,3n -k4,4n 表示以.为分隔符，按IP的四个段分别做数字排序

更高效的版本（针对日志格式优化） #

观察你的日志，IP总是跟在lease后面，所以可以直接定位字段，不用遍历所有字段：

awk '/lease 100\.64\./{a[$7]++} END{for(x in a) print x,a[x]}' logfile | sort -t. -k1,1n -k2,2n -k3,3n -k4,4n

• 日志中lease后面的第1个字段就是IP（即$7，可以根据实际日志字段数调整），直接取这个字段统计，效率更高

内容的提问来源于stack exchange，提问作者John