升级Python/Numpy/great-expectations后Int64类型校验异常问题

环境版本 #

• Python版本：3.8.10（从3.6升级）
• Numpy版本：1.21.5（从1.19.5升级）
• great-expectations版本：0.16.16（从0.13.11升级）

问题现象 #

升级环境后，使用great-expectations校验DataFrame时，NumMonths列出现实际数据类型与预期数据类型不匹配的异常，相同代码在旧版本环境可正常运行。

相关代码 #

DataFrame构造代码 #

data = {'Ids': [f'A{i}' for i in range(1, 11)],
        'NumMonths': [3, 6, 9, 12, 3, 6, 9, 12, 3, 6]}
df = pd.DataFrame(data)
df['NumMonths'] = df['NumMonths']

great-expectations校验规则 #

{
    "data_asset_type": "Dataset",
    "expectation_suite_name": "testsuite",
    "expectations": [
        {
            "expectation_type": "expect_table_columns_to_match_set",
            "kwargs": {
                "column_set": [
                    "Ids",
                    "NumMonths"
                ],
                "exact_match": true
            },
            "meta": {
                "severity": "critical"
            }
        },
        {
            "expectation_type": "expect_select_column_values_to_be_unique_within_record",
            "kwargs": {
                "column_list": [
                    "Ids"
                ]
            },
            "meta": {
                "severity": "critical"
            }
        },
        {
            "expectation_type": "expect_column_values_to_be_in_type_list",
            "kwargs": {
                "column": "Ids",
                "type_list": [
                    "str"
                ]
            },
            "meta": {
                "severity": "critical"
            }
        },
        {
            "expectation_type": "expect_column_values_to_be_in_type_list",
            "kwargs": {
                "column": "NumMonths",
                "type_list": [
                    "Int32",
                    "Int64"
                ]
            },
            "meta": {
                "severity": "critical"
            }
        },
        {
            "expectation_type": "expect_column_values_to_be_between",
            "kwargs": {
                "column": "NumMonths",
                "min_value": 3,
                "max_value": 12
            },
            "meta": {
                "severity": "warning"
            }
        },
        {
            "expectation_type": "expect_column_values_to_not_be_null",
            "kwargs": {
                "column": "Ids",
                "mostly": 1
            },
            "meta": {
                "severity": "critical"
            }
        },
        {
            "expectation_type": "expect_column_values_to_not_be_null",
            "kwargs": {
                "column": "NumMonths",
                "mostly": 1
            },
            "meta": {
                "severity": "critical"
            }
        }
    ],
    "meta": {
        "great_expectations_version": "0.16.16"
    }
}

排查过程 #

• Numpy 1.20.0版本存在数据类型变更，这是导致类型识别异常的根源
• great-expectations源码中针对Numpy dtype变更有相关处理逻辑，但仍存在适配问题
• 尝试用astype()、convert_dtypes()将NumMonths列转为Int64类型，但进入great-expectations的validate()方法后，该列仍被识别为int64类型

临时解决方法 #

将Numpy版本回退至1.19.5后，代码可正常运行。

内容的提问来源于stack exchange，提问作者Hussain Madarwala