如何将行存储的多测点时空坐标文本数据转换为列存储长表格式？

核心优化思路 #

• 放弃全量加载文件到内存的逻辑，采用逐行读取、逐行转换、即时写入的处理模式，内存占用仅和单行数据大小有关，和文件总大小无关，可完美处理TB级超大文件
• 不需要拆分存储163个独立列表，每读取一行就拆分字段按3个一组遍历坐标，直接生成目标格式行写入，无冗余内存开销

实现代码 #

基础版本（逻辑简单，适配绝大多数场景）：

# 配置常量：坐标点总数量
POINT_TOTAL = 54

with open("源文件路径.txt", "r", encoding="utf-8") as infile, open("输出文件路径.txt", "w", encoding="utf-8") as outfile:
    # 写入目标格式表头
    outfile.write("Time\tx\ty\tz\n\n")
    # 跳过源文件第一行表头
    next(infile)
    
    for line in infile:
        line = line.strip()
        # 跳过空行
        if not line:
            continue
        fields = line.split()
        # 取当前行的时间戳
        current_time = fields[0]
        # 遍历所有坐标点，每3个字段为一组x/y/z
        for i in range(POINT_TOTAL):
            idx = 1 + i * 3
            outfile.write(f"{current_time}\t{fields[idx]}\t{fields[idx+1]}\t{fields[idx+2]}\n")

可选性能优化 #

如果文件体积极大，单条写入IO开销较高，可以增加行缓冲区，攒够指定行数再统一写入，进一步提升处理速度：

POINT_TOTAL = 54
# 缓冲区大小可根据内存情况调整，数值越大IO次数越少、速度越快
BUFFER_LINES = 10000

with open("源文件路径.txt", "r", encoding="utf-8") as infile, open("输出文件路径.txt", "w", encoding="utf-8") as outfile:
    outfile.write("Time\tx\ty\tz\n\n")
    next(infile)
    write_buffer = []
    
    for line in infile:
        line = line.strip()
        if not line:
            continue
        fields = line.split()
        current_time = fields[0]
        for i in range(POINT_TOTAL):
            idx = 1 + i * 3
            write_buffer.append(f"{current_time}\t{fields[idx]}\t{fields[idx+1]}\t{fields[idx+2]}\n")
        # 缓冲区满则写入并清空
        if len(write_buffer) >= BUFFER_LINES:
            outfile.writelines(write_buffer)
            write_buffer.clear()
    # 写入最后残留的缓冲区内容
    if write_buffer:
        outfile.writelines(write_buffer)

内容的提问来源于stack exchange，提问作者Lois