Python复杂服务器日志解析与结构化数据生成需求

需求梳理 #

• 跳过含ErrorDown字段的日志条目
• 将Lined字段转换为数值或数组类型
• 拆分obtained字段为obtained（已获取值）和obtained_Top（上限值）
• 用生成器流式处理数据，实现高效解析

实现方案 #

1. 通用正则匹配与字段处理逻辑 #

用正则匹配日志中的键值对，针对特殊字段做针对性处理，同时通过生成器实现流式解析：

import re
import json

# 匹配键值对的正则，适配带空格的值，通过正向预查分割
KEY_VALUE_PATTERN = re.compile(r'(\w+):\s*(.*?)(?=\s+\w+:|$)')

def parse_single_log(line):
    # 直接跳过包含ErrorDown的行
    if 'ErrorDown:' in line:
        return None
    
    parsed_data = {}
    matches = KEY_VALUE_PATTERN.findall(line.strip())
    
    for key, raw_value in matches:
        if key == 'Lined':
            # 处理Lined：逗号分隔则转整数数组，否则转单个整数
            if ',' in raw_value:
                parsed_data[key] = [int(num.strip()) for num in raw_value.split(',')]
            else:
                parsed_data[key] = int(raw_value) if raw_value.isdigit() else raw_value
        elif key == 'obtained':
            # 拆分obtained：按/分割为实际值和上限值
            if '/' in raw_value:
                current, top = raw_value.split('/')
                parsed_data['obtained'] = int(current.strip())
                parsed_data['obtained_Top'] = int(top.strip())
            else:
                # 兼容无上限值的情况
                parsed_data['obtained'] = int(raw_value.strip())
                parsed_data['obtained_Top'] = None
        else:
            # 通用字段类型转换
            if raw_value.isdigit():
                parsed_data[key] = int(raw_value)
            elif raw_value.lower() in ('true', 'false'):
                parsed_data[key] = raw_value.lower() == 'true'
            else:
                parsed_data[key] = raw_value
    
    return parsed_data

# 生成器：流式处理日志，输出JSON字符串
def log_to_json_stream(log_lines):
    for line in log_lines:
        line = line.strip()
        if not line:
            continue
        result = parse_single_log(line)
        if result:
            yield json.dumps(result)

2. 使用示例 #

测试日志输入

Lined: 123,456 ErrorDown: true obtained: 80/100 Name: ServerA
Lined: 789 obtained: 95/100 Name: ServerB
obtained: 50 Name: ServerC

调用代码

# 模拟日志流（实际可替换为文件读取器）
sample_logs = [
    "Lined: 123,456 ErrorDown: true obtained: 80/100 Name: ServerA",
    "Lined: 789 obtained: 95/100 Name: ServerB",
    "obtained: 50 Name: ServerC"
]

# 流式输出JSON
for json_str in log_to_json_stream(sample_logs):
    print(json_str)

输出结果

{"Lined": [123, 456], "obtained": 80, "obtained_Top": 100, "Name": "ServerA"}
{"Lined": 789, "obtained": 95, "obtained_Top": 100, "Name": "ServerB"}
{"obtained": 50, "obtained_Top": null, "Name": "ServerC"}

关键细节说明 #

• 正则(\w+):\s*(.*?)(?=\s+\w+:|$)通过正向预查(?=\s+\w+:|$)确保值中包含空格时也能正确分割键值对
• 生成器逐行处理，适合GB级大日志文件，避免一次性加载全部内容导致内存溢出
• Lined字段自动判断格式，兼容单值和数组两种情况
• obtained拆分逻辑兼容带上限和不带上限的场景，提升解析鲁棒性

内容的提问来源于stack exchange，提问作者Juliana Ashok