Polars写入IPC文件体积异常过大问题求助

Polars将数据持久化到磁盘时，生成的IPC文件体积始终异常庞大；而Parquet格式的文件大小与pandas相当，甚至如预期略小。

具体情况：pandas生成的Feather文件约250MB，而Polars生成的IPC文件约2.5GB。多数文件体积会膨胀1.5到10倍，且膨胀幅度似乎与列数相关。

我认为这并非PyArrow的问题，因为pandas生成的文件大小符合预期；也不认为这是Polars-Python的Bug，推测是本地环境问题，希望能得到帮助。

可复现问题的最简代码 #

import os
import pandas as pd
import yfinance
import polars as pl
import subprocess

os.environ["POLARS_VERBOSE"] = "1"

# datetime example
data = yfinance.download(
  tickers="^AXJO ^N225", start="2005-01-01", end="2024-06-01", interval="1d", threads=True, prepost=True
)
data.columns = data.columns.get_level_values(1) + "_" + data.columns.get_level_values(0)
data.index = pd.DatetimeIndex(data.index, name="date_utc").tz_localize(None)
data = data.reset_index()

# both results in datetime[ns] / float64 for all columns
data.to_feather("banana.feather")
pl.DataFrame(data).write_ipc("banana_pl2.feather")

# integer example w larger data
data2 = {
  "vals1": range(50_000),
  "vals2": range(50_000),
  "vals3": range(50_000)
}

# disproving this is a pandas->polars artefact
pd.DataFrame(data2).to_feather("b.feather")
pl.DataFrame(data2).write_ipc("b_pl.feather")

command = "ls -lha | grep feather"
subprocess.Popen(command, shell=True)

文件大小输出 #

-rw-r--r--   1 xxx.xxx  884741199   589K Jun  9 11:41 b.feather
-rw-r--r--   1 xxx.xxx  884741199   1.1M Jun  9 11:41 b_pl.feather
-rw-r--r--   1 xxx.xxx  884741199   338K Jun  9 11:41 banana.feather
-rw-r--r--   1 xxx.xxx  884741199   523K Jun  9 11:41 banana_pl.feather

测试环境配置 #

环境1 #

--------Version info---------
Polars:               0.20.31
Index type:           UInt32
Platform:             macOS-13.6.5-arm64-arm-64bit
Python:               3.10.6 (main, Aug  7 2023, 13:38:39) [Clang 14.0.3 (clang-1403.0.22.14.1)]

----Optional dependencies----
adbc_driver_manager:  <not installed>
cloudpickle:          3.0.0
connectorx:           <not installed>
deltalake:            <not installed>
fastexcel:            <not installed>
fsspec:               <not installed>
gevent:               <not installed>
hvplot:               <not installed>
matplotlib:           3.8.0
nest_asyncio:         1.5.8
numpy:                1.26.0
openpyxl:             3.1.2
pandas:               2.2.2
pyarrow:              16.1.0
pydantic:             <not installed>
pyiceberg:            <not installed>
pyxlsb:               <not installed>
sqlalchemy:           2.0.22
torch:                <not installed>
xlsx2csv:             <not installed>
xlsxwriter:           <not installed>

环境2 #

--------Version info---------
Polars:               0.20.19
Index type:           UInt32
Platform:             macOS-13.6.5-arm64-arm-64bit
Python:               3.10.6 (main, Aug  7 2023, 13:38:39) [Clang 14.0.3 (clang-1403.0.22.14.1)]

----Optional dependencies----
adbc_driver_manager:  <not installed>
cloudpickle:          3.0.0
connectorx:           <not installed>
deltalake:            <not installed>
fastexcel:            <not installed>
fsspec:               <not installed>
gevent:               <not installed>
hvplot:               <not installed>
matplotlib:           3.8.0
nest_asyncio:         1.5.8
numpy:                1.26.0
openpyxl:             3.1.2
pandas:               2.1.1
pyarrow:              13.0.0
pydantic:             <not installed>
pyiceberg:            <not installed>
pyxlsb:               <not installed>
sqlalchemy:           2.0.22
torch:                <not installed>
xlsx2csv:             <not installed>
xlsxwriter:           <not installed>

内容的提问来源于stack exchange，提问作者md1