Pandas、NumPy、Polars等工具多文件追加性能对比及底层原理咨询

测试背景 #

为适配业务需求，评估了NumPy、Pandas、Polars、CSV模块及标准Python方法的各类解析器性能，将多个.txt文件合并为单个大文件。合并后的文件可存入内存（约46MB），包含100万行数据，每行含4个浮点型数值，以逗号分隔。此前多篇文章提及当行数超过50万时Pandas性能更优，因此开展本次基准测试。

测试数据样例 #

12978.0,625730956116471.3275,843920.00,0.15
242553.0,875730956800704.2362,1737251.00,0.16
857385.0,891730956926730.5725,1697958.00,0.18
569484.0,258730957100957.8962,3285886.00,0.20
...

测试脚本 #

import sys
import csv as csv
import numpy as np
import pandas as pd
import polars as pl
import time
import os

# total files to append (file_0.txt,file_1.txt,...file_99.txt)
tot_files = 100
in_fl = 'file.txt'

# method-0 (pandas)
print('****Pandas****')
t_start0 = time.time()
fl_out = 'fl_pd.txt'
ln_pd = 0

with open(fl_out,'w') as f_w:
    pass

for fl_id in range(tot_files):
    rd_fl = f'{in_fl[:-4]}_{fl_id}.txt'
    rd_pd = pd.read_csv(rd_fl, chunksize=None, skiprows=2, usecols=[0, 1, 2, 3],
                        names=['var0', 'var1', 'var2', 'var3'])
    rd_pd.to_csv(fl_out, mode='a', index=False, header=False)
    ln_pd += rd_pd.shape[0]  # total rows read

print(f'*Total rows check: {ln_pd}')
print(f'*Time Taken-Pandas: {round(time.time() - t_start0, 1)} s\n')

# method-1 (numpy)
print('****NumPy****')
t_start1 = time.time()
fl_out = 'fl_np.txt'
ln_np = 0

with open(fl_out, 'w') as f_w:
    for fl_id in range(tot_files):
        rd_fl = f'{in_fl[:-4]}_{fl_id}.txt'
        events_np = np.loadtxt(rd_fl, delimiter=',', skiprows=2)
        np.savetxt(f_w, events_np, delimiter=',', fmt='%1.1f,%1.4f,%1.2f,%1.2f')
        ln_np += events_np.shape[0]

print(f'*Total rows check: {ln_np}')
print(f'*Time Taken-NumPy: {round(time.time() - t_start1, 1)} s\n')

# method-2 (python; row-by-row)
print('****Python (RowByRow)****')
t_start2 = time.time()
fl_out = 'fl_rBr.txt'
rBr_cnt = 0

with open(fl_out, 'w') as f_w:
    for fl_id in range(tot_files):
        rd_fl = f'{in_fl[:-4]}_{fl_id}.txt'

        with open(rd_fl, 'r') as f_r:
            for ln in f_r:
                f_w.writelines(ln)
                rBr_cnt += 1

print(f'*Total rows check: {rBr_cnt}')
print(f'*Time Taken-rBr: {round(time.time() - t_start2, 1)} s\n')

# method-3 (python; infile)
print('****Python (inFile)****')
t_start3 = time.time()
fl_out = 'fl_inFile.txt'

with open(fl_out, 'w') as outfile:
    for fl_id in range(tot_files):
        rd_fl = f'{in_fl[:-4]}_{fl_id}.txt'
        with open(rd_fl) as infile:
            outfile.write(infile.read())

print(f'*Time Taken-inFile: {round(time.time() - t_start3, 1)} s')

with open(fl_out, 'r') as f_r:
    inF_cnt = sum(1 for line in f_r)
print(f'*Total rows check: {inF_cnt}\n')

# method-4 (polars)
print('****Polars****')
t_start4 = time.time()
pls_cnt = 0
fl_out = 'fl_pl.txt'

with open(fl_out, mode="w") as f_w:
    for fl_id in range(tot_files):
        rd_fl = f'{in_fl[:-4]}_{fl_id}.txt'
        reader = pl.read_csv(rd_fl, separator=",", skip_rows=2, dtypes=[pl.Float64, pl.Float64, pl.Float64, pl.Float64])
        pls_cnt += reader.shape[0]
        reader.write_csv(f_w, float_precision=4)

print(f'*Time Taken-Polars: {round(time.time() - t_start4, 1)} s')
print(f'*Total rows check: {pls_cnt}\n')

# method-4 (csv module)
print('****Python (csv)****')
t_start5 = time.time()
fl_out = 'fl_csv.txt'
with open(fl_out, 'w', newline="\n") as csvfile:
    csvwriter = csv.writer(csvfile)
    for fl_id in range(tot_files):
        rd_fl = f'{in_fl[:-4]}_{fl_id}.txt'

        with open(rd_fl) as csv_file:
            csv_reader = csv.reader(csv_file, delimiter=',')
            csvwriter.writerows(csv_reader)

print(f'*Time Taken-csv: {round(time.time() - t_start5, 1)} s')

性能测试结果 #

合并100个文件时各解析器的耗时如下，其中Polars性能表现最优：

*Time taken-Pandas: 945.7 s
*Time taken-NumPy: 1055.8 s
*Time taken-rBr: 3795.0 s
*Time taken-inFile: 155.8 s
*Time taken-Polars: 67.4 s
*Time taken-csv: 606.7 s

性能差异的底层原理分析 #

1. Polars 为何最快 #

Polars基于Rust编写，核心逻辑均为编译后的原生代码，不受Python GIL（全局解释器锁）限制，可充分利用多核CPU并行处理。CSV读写阶段采用矢量化处理+批量IO：

• 读取时直接将文件数据加载到内存缓冲区，用原生代码快速解析分隔符、转换数值类型，避免了Python对象与原生数据的频繁切换；
• 写入时批量处理数据，减少IO系统调用次数，浮点格式化逻辑同样由原生代码实现，效率远高于Python层面的格式化操作。
  另外，Polars的内存模型紧凑，浮点数据存储在连续内存块中，降低了内存开销与访问延迟。

2. Python inFile 方法次快 #

该方法直接调用操作系统文件接口，一次性读取整个文件内容再写入，本质是操作系统层面的批量IO，跳过了Python层面逐行解析或类型转换的开销。但它仅做文件内容复制，未处理数据（比如测试中未跳过前两行），这也是其速度快的原因之一。

3. Pandas 性能不如预期 #

Pandas虽底层有C优化，但DataFrame带有大量元数据与索引信息，处理100个小文件时，每次read_csv和to_csv都会初始化DataFrame、处理索引与列信息，这些额外开销累加后非常可观。此外，Pandas的CSV解析器在小文件场景下无法发挥并行优势，追加写入时的IO操作也增加了系统调用开销。

4. NumPy 速度较慢 #

np.loadtxt的CSV解析逻辑虽有C优化，但核心逐行解析仍在Python层面循环，效率远低于专业CSV解析器。np.savetxt格式化浮点数据时需逐个元素处理，加上数组写入时的类型检查与格式转换开销，导致整体速度变慢。

5. 原生Python逐行（RowByRow）最慢 #

逐行读写需在Python层面循环处理每一行，每次循环都会触发解释器指令执行，受GIL限制无法并行。且逐行写入会产生大量IO系统调用，操作系统IO缓存无法充分发挥作用，效率极低。

6. Python csv 模块性能中等 #

csv模块由C实现，解析效率高于纯Python逐行处理，但仍需在Python层面处理每行元素，writerows的批量写入仍存在Python对象与C层面的转换开销，且无并行处理能力，因此速度不如Polars和inFile方法。

内容的提问来源于stack exchange，提问作者nuki