Polars读CSV内存优化咨询及Pandas与Polars读写性能对比

测试结论与疑问 #

• 即使使用Pandas 2，采用numpy后端的版本也比pyarrow后端更快，这完全出乎意料——原本以为pyarrow会让Pandas 2提速，结果反而更慢；Polars的读取速度优势明显，是Pandas 2的3倍。
• Polars的内存占用与Pandas 2相近，约为753MB。
• 若通过pyarrow引擎将CSV转为Parquet文件，Pandas 2与Polars的读取速度相当，甚至Pandas略快，这让我考虑是否继续使用Pandas，仅做CSV转Parquet的处理即可。

我正在使用Dash开发包含大量热图的仪表盘，内存占用优化对我至关重要。请问如何降低Polars read_csv()的内存占用？

测试代码及结果 #

1. 使用Pandas 2 + pyarrow读取CSV #

timer_start=timeit.default_timer()
df_pyarrow=pd.read_csv('input\\'+filename,parse_dates=True,sep='\t',engine='pyarrow')  
timer_end=timeit.default_timer()

timer_minutes=(timer_end-timer_start)/60
timer_seconds=(timer_end-timer_start)

print(f'Time took to Finish reading file is {timer_seconds:.1f} seconds')
print(f'Time took to Finish reading file is {timer_minutes:.2f} minutes')

# 输出结果
Time took to Finish reading file is 206.8 seconds
Time took to Finish reading file is 3.45 minutes

df_pyarrow.info()

# 输出结果
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3675 entries, 0 to 3674
Columns: 26903 entries, Depth to 2023-03-09 00:06:28
dtypes: float64(26903)
memory usage: 754.3 MB

2. 使用无pyarrow的Pandas 2读取CSV #

timer_start=timeit.default_timer()
df_dts2=pd.read_csv('input\\'+filename,parse_dates=True,sep='\t')  
timer_end=timeit.default_timer()

timer_minutes=(timer_end-timer_start)/60
timer_seconds=(timer_end-timer_start)

print(f'Time took to Finish reading file is {timer_seconds:.1f} seconds')
print(f'Time took to Finish reading file is {timer_minutes:.2f} minutes')

# 输出结果
Time took to Finish reading file is 121.2 seconds
Time took to Finish reading file is 2.02 minutes

df_dts2.info()
# 输出结果
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3675 entries, 0 to 3674
Columns: 26903 entries, Depth to 2023-03-09 00:06:28
dtypes: float64(26903)
memory usage: 754.3 MB

3. 使用Polars读取CSV #

import polars as pl
timer_start=timeit.default_timer()
df_dts2=pl.read_csv('input\\'+filename,separator='\t')  
timer_end=timeit.default_timer()

timer_minutes=(timer_end-timer_start)/60
timer_seconds=(timer_end-timer_start)

print(f'Time took to Finish reading file is {timer_seconds:.1f} seconds')
print(f'Time took to Finish reading file is {timer_minutes:.2f} minutes')

# 输出结果
Time took to Finish reading file is 37.7 seconds
Time took to Finish reading file is 0.63 minutes

df_dts2.estimated_size("mb")
# 输出结果
763.4694671630859

4. 读取Parquet文件 #

案例：使用Pandas 2

timer_start=timeit.default_timer()
df_parquet_pyarrow=pd.read_parquet('input\\'+filename_parquet_pyarrow, engine='pyarrow')
timer_end=timeit.default_timer()
timer_minutes=(timer_end-timer_start)/60
timer_seconds=(timer_end-timer_start)

print(f'Time took to Finish reading file is {timer_seconds:.1f} seconds')
print(f'Time took to Finish reading file is {timer_minutes:.2f} minutes')

# 输出结果
Time took to Finish reading file is 13.0 seconds
Time took to Finish reading file is 0.22 minutes

案例：使用Polars

import polars as pl
timer_start=timeit.default_timer()
df_polar_parquet=pl.read_parquet('input\\'+filename_parquet_pyarrow)  
timer_end=timeit.default_timer()

timer_minutes=(timer_end-timer_start)/60
timer_seconds=(timer_end-timer_start)

print(f'Time took to Finish reading file is {timer_seconds:.1f} seconds')
print(f'Time took to Finish reading file is {timer_minutes:.2f} minutes')

# 输出结果
Time took to Finish reading file is 15.7 seconds
Time took to Finish reading file is 0.26 minutes

内容的提问来源于stack exchange，提问作者roudan