Polars与pandera(0.19.0b3)空值转换错误排查求助

问题场景 #

使用pandera 0.19.0b3结合Polars 0.20.23做数据验证时，遇到异常：当数据中存在nullable=True的列（如示例里的extract_date包含None值），同时其他列存在验证失败数据（如case_id含无法转int的字符串"abc"），会抛出polars.exceptions.ComputeError，而非预期的SchemaErrors提示。

当case_id全为可转换为int的字符串时，验证正常；只要包含非法值，就触发异常。

安装命令 #

pip install pre 'pandera[polars]'

复现代码 #

import pandera.polars as pa
import polars as pl
from datetime import date
import json

class CaseSchema(pa.DataFrameModel):
    case_id: int = pa.Field(nullable=False, unique=True, coerce=True)
    gdwh_portfolio_id: str = pa.Field(nullable=False, unique=True, coerce=True)
    extract_date: date = pa.Field(nullable=True, coerce=True)

    class Config:
        drop_invalid_rows = True

invalid_lf = pl.DataFrame({
    #"case_id": ["1", "2", "3"],  # 此场景验证正常
    "case_id": ["1", "2", "abc"],  # 此场景触发异常
    "gdwh_portfolio_id": ["d", "e", "f"],
    "extract_date": [date(2024,1,1), date(2024,1,2), None]
})

try:
    CaseSchema.validate(invalid_lf, lazy=True)
except pa.errors.SchemaErrors as e:
        print(json.dumps(e.message, indent=4))

错误信息 #

核心报错：

polars.exceptions.ComputeError: conversion from `struct[3]` to `str` failed in column 'failure_case' for 1 out of 1 values: [{"abc","f",null}]

完整报错栈：

> Traceback (most recent call last):
>   File "<frozen runpy>", line 198, in _run_module_as_main
>   File "<frozen runpy>", line 88, in _run_code
>   File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/erehoba-acc-payments-req/code/Users/ourrehman/dna-payments-and-accounts/data_validation/test.py", line 22, in <module>
>     CaseSchema.validate(invalid_lf, lazy=True)
>   File "/anaconda/envs/pandera-polars/lib/python3.11/site-packages/pandera/api/dataframe/model.py", line 289, in validate
>     cls.to_schema().validate(
>   File "/anaconda/envs/pandera-polars/lib/python3.11/site-packages/pandera/api/polars/container.py", line 58, in validate
>     output = self.get_backend(check_obj).validate(
>              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/anaconda/envs/pandera-polars/lib/python3.11/site-packages/pandera/backends/polars/container.py", line 65, in validate
>     check_obj = parser(check_obj, *args)
>                 ^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/anaconda/envs/pandera-polars/lib/python3.11/site-packages/pandera/backends/polars/container.py", line 398, in coerce_dtype
>     check_obj = self._coerce_dtype_helper(check_obj, schema)
>                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/anaconda/envs/pandera-polars/lib/python3.11/site-packages/pandera/backends/polars/container.py", line 486, in _coerce_dtype_helper
>     raise SchemaErrors(
>           ^^^^^^^^^^^^^
>   File "/anaconda/envs/pandera-polars/lib/python3.11/site-packages/pandera/errors.py", line 183, in __init__
>     ).failure_cases_metadata(schema.name, schema_errors)
>       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/anaconda/envs/pandera-polars/lib/python3.11/site-packages/pandera/backends/polars/base.py", line 173, in failure_cases_metadata
>     ).cast(
>       ^^^^^
>   File "/anaconda/envs/pandera-polars/lib/python3.11/site-packages/polars/dataframe/frame.py", line 6624, in cast
>     return self.lazy().cast(dtypes, strict=strict).collect(_eager=True)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/anaconda/envs/pandera-polars/lib/python3.11/site-packages/polars/lazyframe/frame.py", line 1810, in collect
>     return wrap_df(ldf.collect())
>                    ^^^^^^^^^^^^^
> polars.exceptions.ComputeError: conversion from `struct[3]` to `str` failed in column 'failure_case' for 1 out of 1 values: [{"abc","f",null}]

期望行为 #

验证流程应正常处理含空值的nullable=True列，当存在验证失败数据时，正确抛出SchemaErrors并输出错误详情，而非触发Polars类型转换异常。

解决办法 #

该问题是pandera 0.19.0b3的已知bug，根源是失败行包含null值时，pandera生成错误详情的failure_case列时，无法将含null的struct转换为字符串。

方案1：捕获ComputeError并手动提取信息 #

在异常处理中新增对ComputeError的捕获，提取关键错误内容：

import pandera.polars as pa
import polars as pl
from datetime import date
import json
from polars.exceptions import ComputeError

class CaseSchema(pa.DataFrameModel):
    case_id: int = pa.Field(nullable=False, unique=True, coerce=True)
    gdwh_portfolio_id: str = pa.Field(nullable=False, unique=True, coerce=True)
    extract_date: date = pa.Field(nullable=True, coerce=True)

    class Config:
        drop_invalid_rows = True

invalid_lf = pl.DataFrame({
    "case_id": ["1", "2", "abc"],
    "gdwh_portfolio_id": ["d", "e", "f"],
    "extract_date": [date(2024,1,1), date(2024,1,2), None]
})

try:
    CaseSchema.validate(invalid_lf, lazy=True)
except pa.errors.SchemaErrors as e:
    print(json.dumps(e.message, indent=4))
except ComputeError as e:
    error_msg = str(e).split(":")[-1].strip()
    print(f"验证失败，错误行：{error_msg}")

方案2：升级pandera版本 #

该bug已在后续版本修复，直接升级到最新正式版即可：

pip install --upgrade 'pandera[polars]'

方案3：临时禁用lazy模式 #

改用非lazy验证模式可规避该问题：

CaseSchema.validate(invalid_lf, lazy=False)

内容的提问来源于stack exchange，提问作者Obiii