PyArrow CSV转Parquet时无效UTF8报错定位错误行及处理方法问询

1. 定位具体报错行 #

PyArrow的read_csv函数提供了invalid_row_handler参数，可以自定义处理异常行并收集行号，示例代码如下：

import pyarrow as pa
import pyarrow.csv as csv

# 存储异常行号
invalid_rows = []

def handle_invalid_row(row):
    # row.number对应CSV的行号，默认从1开始计数（包含表头行）
    invalid_rows.append(row.number)
    # 跳过当前异常行继续读取
    return pa.csv.InvalidRowResult.SKIP

# 配置解析选项
parse_options = csv.ParseOptions(invalid_row_handler=handle_invalid_row)
# 读取CSV
table = csv.read_csv("your_large_file.csv", parse_options=parse_options)

# 输出所有异常行号
print("异常行号：", invalid_rows)

2. 直接替换无效UTF-8单元格为NULL #

方案1：高版本PyArrow（12.0+）直接配置转换选项 #

通过ConvertOptions的invalid_utf8_handling参数处理无效字符，后续可按需转NULL：

import pyarrow.csv as csv
import pyarrow.compute as pc

convert_options = csv.ConvertOptions(
    # 无效UTF-8字符替换为�，避免读取报错
    invalid_utf8_handling="replace"
)
table = csv.read_csv("your_large_file.csv", convert_options=convert_options)

# 可选：将含无效字符的单元格转为NULL
col_index = 9 # 对应报错的第10列，索引从0开始
col = table.column(col_index)
is_valid_utf8 = pc.utf8_is_valid(col)
clean_col = pc.if_else(is_valid_utf8, col, None)
table = table.set_column(col_index, table.field(col_index).name, clean_col)

方案2：低版本PyArrow兼容处理 #

先将报错列按二进制类型读取，再手动转UTF-8并替换无效值：

import pyarrow as pa
import pyarrow.csv as csv
import pyarrow.compute as pc

convert_options = csv.ConvertOptions(
    # 把报错的第10列先按二进制读取，避免编码报错
    column_types={"your_column_name": pa.binary()}
)
table = csv.read_csv("your_large_file.csv", convert_options=convert_options)

# 转UTF-8，无效字符直接置为NULL
col_index = 9
clean_col = pc.utf8_replace_invalid(table.column(col_index), replacement_char=None)
table = table.set_column(col_index, "your_column_name", clean_col)

补充：如果确认文件是非UTF-8编码 #

可以先在读取选项中指定正确编码，从根源解决报错：

read_options = csv.ReadOptions(encoding="gbk") # 可替换为GBK、latin-1等实际编码
table = csv.read_csv("your_large_file.csv", read_options=read_options)

内容的提问来源于stack exchange，提问作者Avinash Raj