日志分析中基于日期范围过滤日志的函数异常问题排查

我正在开展日志分析工作，需先提取日志文件中的日期，再以此定义起始日期与结束日期，筛选该时间范围内的日志内容。目前已通过正则表达式成功提取日期，但filter_log_entries函数未按预期过滤日志内容。

原函数代码 #

@staticmethod
def filter_log_entries(log_content, start_date, end_date):
    start_datetime = datetime.strptime(start_date, '%d/%b/%Y').replace(tzinfo=timezone.utc)
    end_datetime = datetime.strptime(end_date, '%d/%b/%Y').replace(tzinfo=timezone.utc)

    # Adjust end_datetime to include the entire end day
    end_datetime = end_datetime + timedelta(days=1) - timedelta(seconds=1)

    log_entry_pattern = re.compile(r'\[(\d{2}/[A-Za-z]{3}/\d{4}:\d{2}:\d{2}:\d{2} [+-]\d{4})\]')
    filtered_entries = []

    for line in log_content.split('\n'):
        match = log_entry_pattern.search(line)
        if match:
            entry_datetime_str = match.group(1)
            try:
                entry_datetime = datetime.strptime(entry_datetime_str, '%d/%b/%Y:%H:%M:%S %z')
                if start_datetime <= entry_datetime <= end_datetime:
                    filtered_entries.append(line)
            except ValueError:
                st.write(f"Date parsing error for line: {line}")

    filtered_log_content = "\n".join(filtered_entries)

    return filtered_log_content

场景说明 #

• 日志日期格式：[17/May/2015:10:05:03 +0000]
• 日志最晚日期：[20/May/2015:10:05:03 +0000]
• 目标：筛选2015年5月17日至2015年5月18日的所有日志

示例日志内容 #

83.149.9.216 - - [17/May/2015:10:05:03 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:43 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-dashboard3.png HTTP/1.1" 200 171717 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:47 +0000] "GET /presentations/logstash-monitorama-2013/plugin/highlight/highlight.js HTTP/1.1" 200 26185 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:12 +0000] "GET /presentations/logstash-monitorama-2013/plugin/zoom-js/zoom.js HTTP/1.1" 200 7697 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:07 +0000] "GET /presentations/logstash-monitorama-2013/plugin/notes/notes.js HTTP/1.1" 200 2892 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:34 +0000] "GET /presentations/logstash-monitorama-2013/images/sad-medic.png HTTP/1.1" 200 430406 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:57 +0000] "GET /presentations/logstash-monitorama-2013/css/fonts/Roboto-Bold.ttf HTTP/1.1" 200 38720 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:50 +0000] "GET /presentations/logstash-monitorama-2013/css/fonts/Roboto-Regular.ttf HTTP/1.1" 200 41820 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:24 +0000] "GET /presentations/logstash-monitorama-2013/images/frontend-response-codes.png HTTP/1.1" 200 52878 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:50 +0000] 

问题分析与修复 #

原函数核心问题在于结束时间计算方式不够严谨：通过timedelta(days=1) - timedelta(seconds=1)获取结束日最后一秒，易出现时区或精度误差，且判断逻辑冗余。

正确做法是将结束时间设为结束日期的次日0点，用start_datetime <= entry_datetime < end_datetime筛选，完整包含起始日到结束日的所有日志，无需处理秒级计算。

修复后的函数代码：

@staticmethod
def filter_log_entries(log_content, start_date, end_date):
    # 解析起始日期为UTC时区的0点
    start_datetime = datetime.strptime(start_date, '%d/%b/%Y').replace(tzinfo=timezone.utc)
    # 解析结束日期，加1天得到次日0点作为过滤上限（不包含）
    end_datetime = datetime.strptime(end_date, '%d/%b/%Y').replace(tzinfo=timezone.utc) + timedelta(days=1)

    log_entry_pattern = re.compile(r'\[(\d{2}/[A-Za-z]{3}/\d{4}:\d{2}:\d{2}:\d{2} [+-]\d{4})\]')
    filtered_entries = []

    for line in log_content.split('\n'):
        # 跳过空行
        if not line.strip():
            continue
        match = log_entry_pattern.search(line)
        if match:
            entry_datetime_str = match.group(1)
            try:
                entry_datetime = datetime.strptime(entry_datetime_str, '%d/%b/%Y:%H:%M:%S %z')
                # 包含起始日，且小于结束日次日0点（即包含整个结束日）
                if start_datetime <= entry_datetime < end_datetime:
                    filtered_entries.append(line)
            except ValueError:
                st.write(f"日期解析错误，日志行：{line}")

    filtered_log_content = "\n".join(filtered_entries)
    return filtered_log_content

关键改动点 #

1. 结束时间调整：改为结束日期次日0点，避免秒级计算误差，逻辑更直观。
2. 判断条件优化：使用start_datetime <= entry_datetime < end_datetime，确保完整覆盖目标时间段。
3. 空行处理：增加空行跳过逻辑，避免无效空行混入结果。

内容的提问来源于stack exchange，提问作者Khawar Islam