如何用Python拆分多分隔符日志行并优化数据提取流程？

问题描述 #

日志数据示例 #

date Mon Jan 4 15:59:21.129 2021

base hex  timestamps absolute

no internal events logged

// version 13.0.0

//545285.973861 previous log file: Myfile_0.asc

// Measurement UUID: 4520e127-a0b6-48d2-9e23-2588160af285

545333.620639 LoggingString := "Log,11:28 PM, Sunday, January 10, 2021,11:28:17.4,34.72,12,0.01058,11.99,0.01077,12,0.01127,11.99,0.01142,11.76,0.1053,11.99,0.01076,11.96,0.01092,2.516,0,2,OM_2_1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0"

545335.691676 LoggingString := "Log,11:28 PM, Sunday, January 10, 2021,11:28:19.5,34.61,12,0.01058,11.99,0.01072,11.99,0.01127,11.99,0.01139,11.87,0.1118,12.01,0.01046,11.99,0.01145,2.581,0,2,OM_2_1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0"

545337.796715 LoggingString := "Log,11:28 PM, Sunday, January 10, 2021,11:28:21.6,34.52,11.99,0.0106,11.99,0.01077,11.99,0.01151,11.99,0.01139,11.72,0.1081,12,0.0109,11.96,0.01107,2.543,0,2,OM_2_1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0"

545339.919752 LoggingString := "Log,11:28 PM, Sunday, January 10, 2021,11:28:23.7,34.41,12,0.01082,11.99,0.01104,11.99,0.01156,11.99,0.01164,11.62,0.1042,11.99,0.01105,11.96,0.01126,2.596,0,2,OM_2_1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0"

核心需求与疑问 #

1. 日志行拆分：需将每行按空格、逗号拆分，得到类似 ['545333.620639','LoggingString :=', 'Log', ..., 2021, '11:28:17.4', 34.72 ...] 的列表，方便提取11:28:17.4之后的关键数据。
2. readlines()存储方式：调用该方法后，数据是以完整字符串还是单个元素形式存储在列表中？
3. 代码优化需求：
   • 替换硬编码行号，改为检测到// Measurement UUID:后从下一行开始读取数据生成CSV；
   • 实现每行元素的正确拆分；
   • 为np.genfromtxt配置多分隔符支持。

当前实现代码 #

import numpy as np

Testfile = open('C:/Documents/Myfile.asc','r')
Read_data = Testfile.readlines()
count = 0
for line in Read_data:
     count += 1
     if count < 7:        ## counter to start saving data into .csv from 7th line
         print("Line{}: {}".format(count, line.strip()))
     else:
         mydat = np.genfromtxt("C:/Documents/Myfile.asc",skip_header=(count-1),usecols=   (4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23),delimiter=',')
         Data_frame = pd.DataFrame(mydat)
         Data_frame.to_csv("Triall_3.csv",sep=';')
         exit()

解决方案 #

关于readlines()的存储方式 #

readlines()会将文件的每一行作为单个字符串元素存入列表，比如日志第一行date Mon Jan 4 15:59:21.129 2021就是列表中的一个字符串元素，末尾会保留换行符（可通过strip()去除）。

优化后的实现代码 #

提供两种方案，分别适配不同数据量场景：

方案1：纯Python拆分+动态起始行（灵活处理格式）

import csv

# 定义文件路径
input_path = 'C:/Documents/Myfile.asc'
output_path = 'Triall_3.csv'

start_processing = False
processed_rows = []

with open(input_path, 'r') as f:
    for line in f:
        line = line.strip()
        # 跳过空行
        if not line:
            continue
        # 检测到UUID行，下一行开始处理数据
        if line.startswith('// Measurement UUID:'):
            start_processing = True
            continue
        # 处理数据行
        if start_processing:
            # 按空格拆分前3部分（时间戳、LoggingString、:=），保留引号内内容
            parts = line.split(maxsplit=3)
            timestamp = parts[0]
            identifier = f"{parts[1]} {parts[2]}"
            # 拆分引号内的逗号分隔内容
            quoted_content = parts[3].strip('"')
            quoted_parts = quoted_content.split(',')
            # 合并所有部分得到目标列表
            full_row = [timestamp, identifier] + quoted_parts
            processed_rows.append(full_row)

# 写入CSV文件
with open(output_path, 'w', newline='') as csvfile:
    writer = csv.writer(csvfile, delimiter=';')
    writer.writerows(processed_rows)

方案2：结合numpy+pandas的高效处理（大数据量场景）

import numpy as np
import pandas as pd

input_path = 'C:/Documents/Myfile.asc'
output_path = 'Triall_3.csv'

# 第一步：动态查找数据起始行
start_line = 0
with open(input_path, 'r') as f:
    for idx, line in enumerate(f, 1):
        if '// Measurement UUID:' in line:
            start_line = idx + 1
            break

# 第二步：配置numpy多分隔符（匹配空格或逗号）
dat = np.genfromtxt(
    input_path,
    skip_header=start_line,
    delimiter=r'\s+|,',  # 正则匹配一个或多个空格，或逗号
    dtype=None,
    encoding='utf-8'
)

# 第三步：转为DataFrame并写入CSV（如需提取特定列，可添加usecols参数，比如usecols=np.r_[0, 3:25]）
df = pd.DataFrame(dat)
df.to_csv(output_path, sep=';', index=False)

关键说明 #

1. 动态起始行：通过遍历文件定位// Measurement UUID:行，自动设置下一行为数据起始点，彻底替代硬编码行号。
2. 多分隔符处理：
   • 纯Python方案：先拆分空格分隔的前缀部分，再处理引号内的逗号分隔内容，确保拆分准确性；
   • numpy方案：使用正则表达式作为分隔符，同时支持空格和逗号的拆分需求。
3. 行拆分效果：两种方案均可生成目标格式的拆分列表，满足关键数据提取需求。

内容的提问来源于stack exchange，提问作者Sajeev Pillai