Polars转Pandas后PyArrow扩展数组的Reshape问题及解决方法

问题背景 #

使用Polars DataFrame读取文件并循环处理，因需调用一个不可修改的Pandas编写的函数，必须将Polars DataFrame转换为Pandas格式。为提升性能，转换时启用了PyArrow扩展数组：

df = df_pl.to_pandas(use_pyarrow_extension_array = True)

但后续执行reshape操作时出现错误。

报错代码及信息 #

执行以下代码时抛出异常：

X_test = validation.year.array.reshape(1, -1)

错误提示：

<class 'pandas.core.arrays.arrow.array.ArrowExtensionArray'> does not support reshape as backed by a 1D pyarrow.ChunkedArray.

示例代码 #

# using polars
import polars as pl
import pandas as pd

data = {"group1": [1, 1,2,3], "group2": [3, 3,3,2],"group3": [1, 1,4,3], "group4": [3, 3,3,1],"year":[2020,2019,2020,2018]}
df_pl = pl.DataFrame(data)
for group in df_pl.group_by("group1","group2","group3","group4"):
    print(group)
    df_pd = df_pl.to_pandas(use_pyarrow_extension_array = True)
    split_point = len(df_pd) - 1
    df_1, validation = df_pd[0:split_point].copy(), df_pd[split_point:]
    
    test = validation.year.array#.reshape(1, -1)
    print(test)

直接使用Pandas生成的是numpy数组，而Polars转换后得到的是PyArrow扩展数组，该数组不支持reshape操作。

解决方法 #

通过先将PyArrow数组转换为numpy数组，再执行reshape操作即可解决问题：

X_test = validation.year.array.to_numpy().reshape(1, -1)

内容的提问来源于stack exchange，提问作者curiouscoder007