如何用Python统计日志文件中IP对URL的访问次数？

问题说明 #

已能从大型日志文件中提取IP和URL，但当前代码会重复输出同一IP与URL组合，需要实现每个IP访问对应URL的次数统计，要求完全用Python实现，不依赖数据库。

原有代码（存在固定切片提取不可靠、无统计逻辑的问题）：

#!/usr/bin/python3
count = 0
log = open("access.log-20201019", "r")
arr = []
frequency_array = []

for i in log.readlines():
        ip = i[0:14]
        ip2 = ip.split(' ')
        ip3 = ip2[0]
        #print(ip3)
        url =i[53:87]
        url2 = url.split()
        url3 = url2[0]
        print(ip3,url3)

日志片段示例：

66.177.237.17 - - [18/Oct/2020:03:06:07 -0400] "GET /webcam/1/latest.jpeg HTTP/2.0" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36" "-"
158.136.64.65 - - [18/Oct/2020:03:06:07 -0400] "GET /webcam/rwis/littlebay/latest.jpeg HTTP/1.1" 301 169 "-" "curl/7.46.0" "-"
158.136.64.65 - - [18/Oct/2020:03:06:07 -0400] "GET /webcam/rwis/littlebay/latest.jpeg HTTP/1.1" 200 37145 "-" "curl/7.46.0" "-"
112.198.71.230 - - [18/Oct/2020:03:06:09 -0400] "GET /precip/raingauge2.gif HTTP/2.0" 200 10078 "https://www.google.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36" "-"
173.9.45.97 - - [18/Oct/2020:03:06:10 -0400] "GET /NHPR/NHPR_rad_an.gif HTTP/2.0" 200 587317 "-" "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36" "-"
173.9.45.97 - - [18/Oct/2020:03:06:11 -0400] "GET /favicon.ico HTTP/2.0" 200 27877 "https://vortex.plymouth.edu/NHPR/NHPR_rad_an.gif" "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36" "-"
158.136.64.65 - - [18/Oct/2020:03:06:11 -0400] "GET /webcam/1/nograph.1.jpeg HTTP/1.1" 301 169 "-" "curl/7.46.0" "-"
158.136.64.65 - - [18/Oct/2020:03:06:11 -0400] "GET /webcam/1/nograph.1.jpeg HTTP/1.1" 200 242804 "-" "curl/7.46.0" "-"
158.136.64.65 - - [18/Oct/2020:03:06:12 -0400] "GET /webcam/rwis/echolake/latest.jpeg HTTP/1.1" 301 169 "-" "curl/7.46.0" "-"
158.136.64.65 - - [18/Oct/2020:03:06:12 -0400] "GET /webcam/rwis/echolake/latest.jpeg HTTP/1.1" 404 2256 "-" "curl/7.46.0" "-"
158.136.64.65 - - [18/Oct/2020:03:06:14 -0400] "GET /webcam/rwis/lafeyette/latest.jpeg HTTP/1.1" 301 169 "-" "curl/7.46.0" "-"
158.136.64.65 - - [18/Oct/2020:03:06:14 -0400] "GET /webcam/rwis/lafeyette/latest.jpeg HTTP/1.1" 200 36974 "-" "curl/7.46.0" "-"

改进方案 #

核心思路 #

用字典存储统计结果，以(IP, URL)元组作为键，访问次数作为值；同时改用更可靠的方式提取IP和URL，避免固定切片的局限性。

方案1：使用字符串分割提取（简洁高效） #

#!/usr/bin/python3
from collections import defaultdict

# 初始化统计字典，自动处理新键的默认值为0
access_counts = defaultdict(int)

# 用with语句自动管理文件，避免资源泄漏
with open("access.log-20201019", "r") as log_file:
    for line in log_file:
        # 提取IP：取日志行第一个非空白字段
        ip = line.split(' ', 1)[0]
        
        # 提取URL：从引号中拆分出请求行，再取路径部分
        try:
            request_content = line.split('"')[1]
            url = request_content.split()[1]
        except IndexError:
            # 跳过格式异常的日志行
            continue
        
        # 更新访问次数
        access_counts[(ip, url)] += 1

# 遍历输出统计结果
for (ip, url), count in access_counts.items():
    print(f"IP: {ip} | URL: {url} | 访问次数: {count}")

方案2：使用正则表达式提取（更严谨） #

适合日志格式复杂的场景，匹配更精准：

#!/usr/bin/python3
import re
from collections import defaultdict

# 正则匹配模式：匹配开头的IP，以及GET后的URL
log_pattern = r'^(\S+) .*?"GET (\S+)'
access_counts = defaultdict(int)

with open("access.log-20201019", "r") as log_file:
    for line in log_file:
        match_result = re.match(log_pattern, line)
        if match_result:
            ip = match_result.group(1)
            url = match_result.group(2)
            access_counts[(ip, url)] += 1

# 输出结果
for (ip, url), count in access_counts.items():
    print(f"{ip} 访问 {url} 共 {count} 次")

关键改进点 #

1. 替换固定切片：用字符串分割或正则提取IP/URL，适配不同长度的IP和URL，避免提取错误。
2. 统计逻辑实现：用defaultdict(int)简化计数操作，无需手动判断键是否存在；也可改用普通字典，逻辑为if (ip, url) in access_counts: access_counts[(ip, url)] +=1 else: access_counts[(ip, url)] = 1。
3. 文件安全处理：使用with语句自动关闭文件，防止资源泄漏。
4. 异常处理：跳过格式错误的日志行，避免程序崩溃。

内容的提问来源于stack exchange，提问作者user20513526