pyarrow存储含None的18位大整数到parquet时读取值异常

问题现象 #

• 使用Parquet格式存储和读取18位大整数时，读取结果与原始值不一致
• 仅当存储的列表同时包含None和实际数值时触发该问题，列表无None时读写完全正常
• 已排除显示类问题：使用cat、vi等Unix命令直接查看文件，确认不是展示偏差导致的数值异常，例如原始值235313013750949476读取后变为235313013750949472

测试代码 #

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

def get_row_list():
    row_list = []
    row_list.append(None)
    row_list.append(235313013750949476)
    row_list.append(None)
    row_list.append(135313013750949496)
    row_list.append(935313013750949406)
    row_list.append(835313013750949456)
    row_list.append(None)
    row_list.append(None)
    return row_list

def get_row_list_with_no_none():
    row_list = []
    row_list.append(235313013750949476)
    row_list.append(135313013750949496)
    row_list.append(935313013750949406)
    row_list.append(835313013750949456)
    return row_list

def create_parquet(row_list, col_list, parquet_filename):
    df = pd.DataFrame(row_list, columns=col_list)
    schema_field_list = [('tree_id', pa.int64())]
    pa_schema = pa.schema(schema_field_list)
    table = pa.Table.from_pandas(df, pa_schema)
    pq_writer = pq.ParquetWriter(parquet_filename,
                                 schema=pa_schema)
    pq_writer.write_table(table)
    pq_writer.close()
    print("Parquet file [%s] created" % parquet_filename)

def main():
    col_list = ['tree_id']
    # 无空值的测试列表
    row_list = get_row_list_with_no_none()
    print(row_list)
    create_parquet(row_list, col_list, 'without_none.parquet')
    # 混合空值的测试列表
    row_list = get_row_list()
    print(row_list)
    create_parquet(row_list, col_list, 'with_none.parquet')

# ==== 主程序入口 =====
if __name__ == '__main__':
    main()

执行日志 #

python test-parquet.py

[235313013750949476, 135313013750949496, 935313013750949406, 835313013750949456]
Parquet file [without_none.parquet] created
[None, 235313013750949476, None, 135313013750949496, 935313013750949406, 835313013750949456, None, None]
Parquet file [with_none.parquet] created

依赖版本 #

pyarrow                  5.0.0
pandas                   1.1.5
Python 3.6.6

Spark读取验证结果 #

无None的Parquet文件读取结果 #

>>> dfwithoutnone = spark.read.parquet("s3://some-bucket/without_none.parquet/")
>>> dfwithoutnone.count()
4
>>> dfwithoutnone.printSchema()
root
 |-- tree_id: long (nullable = true)

>>> dfwithoutnone.show(10, False)
+------------------+                                                            
|tree_id           |
+------------------+
|235313013750949476|
|135313013750949496|
|935313013750949406|
|835313013750949456|
+------------------+

含None的Parquet文件读取结果 #

>>> df_with_none = spark.read.parquet("s3://some-bucket/with_none.parquet/")
>>> df_with_none.count()
8                                                                               
>>> df_with_none.printSchema()
root
 |-- tree_id: long (nullable = true)

>>> df_with_none.show(10, False)
+------------------+
|tree_id           |
+------------------+
|null              |
|235313013750949472|
|null              |
|135313013750949504|
|935313013750949376|
|835313013750949504|
|null              |
|null              |
+------------------+

解决方案 #

根因 #

你使用的pandas 1.1.5版本原生整数类型不支持缺失值，当传入的列表同时包含整数和None时，pandas会自动将该列隐式转换为float64类型。而64位浮点数的有效精度只有53个二进制位，对应十进制最多可以精确表示16位整数，你用到的18位大整数超出了这个精度范围，转换为float时就已经发生了不可逆的精度丢失，后续即使指定pyarrow按int64存储，保存的也是精度丢失后的值。当列表中没有None时，pandas会直接将列识别为int64类型，没有转换过程，所以读写都正常。

修复方法 #

两种方案任选其一即可：

方案1：指定DataFrame使用pandas可空整数类型

构建DataFrame时，显式指定dtype为Int64（首字母大写，是pandas专门支持缺失值的整数类型）：

df = pd.DataFrame(row_list, columns=col_list, dtype='Int64')

方案2：直接构造pyarrow数组，绕过pandas类型转换

不需要先转DataFrame，直接用Python列表构造pyarrow的int64数组，自动处理None为空值，从根源避免类型转换导致的精度丢失：

def create_parquet(row_list, col_list, parquet_filename):
    arr = pa.array(row_list, type=pa.int64())
    table = pa.Table.from_arrays([arr], names=col_list)
    pq_writer = pq.ParquetWriter(parquet_filename, schema=table.schema)
    pq_writer.write_table(table)
    pq_writer.close()
    print("Parquet file [%s] created" % parquet_filename)

修改后重新生成Parquet文件，再用Spark读取就不会出现精度丢失的问题。

内容的提问来源于stack exchange，提问作者madmatrix