使用UserConfigurableProfiler处理JSON文件触发MetricResolutionError

问题描述 #

尝试使用UserConfigurableProfiler针对特定JSON数据批次生成Expectations，执行代码如下：

import great_expectations as gx
from great_expectations.core.batch import BatchRequest
from great_expectations.profile.user_configurable_profiler import UserConfigurableProfiler

context = gx.get_context()

datasource = context.get_datasource(context.list_datasources()[0]["name"])
data_connector = datasource.data_connectors["sirene_update_s3_connector"].name
data_asset = datasource.get_available_data_asset_names()["sirene_update_s3_connector"][0]
expectation_suite_name = context.list_expectation_suite_names()[0]

batch_request_sirene_update_december_2022 = {
    "datasource_name": datasource.name,
    "data_connector_name": data_connector,
    "data_asset_name": data_asset,
    "data_connector_query": {
        "batch_filter_parameters": {
            "year": "2022",
            "month": "12",
        }
    },
}

validator = context.get_validator(
    batch_request = BatchRequest(**batch_request_sirene_update_december_2022),
    expectation_suite_name = expectation_suite_name,
)

profiler = UserConfigurableProfiler(profile_dataset=validator)

运行时触发错误：

Output exceeds the size limit. Open the full output data in a text editor
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File ~/.pyenv/versions/3.9.6/envs/data_pipelines/lib/python3.9/site-packages/great_expectations/execution_engine/execution_engine.py:595, in ExecutionEngine._process_direct_and_bundled_metric_computation_configurations(self, metric_fn_direct_configurations, metric_fn_bundle_configurations)
    592 try:
    593     resolved_metrics[
    594         metric_computation_configuration.metric_configuration.id
--> 595     ] = metric_computation_configuration.metric_fn(
    596         **metric_computation_configuration.metric_provider_kwargs
    597     )
    598 except Exception as e:

File ~/.pyenv/versions/3.9.6/envs/data_pipelines/lib/python3.9/site-packages/great_expectations/expectations/metrics/metric_provider.py:34, in metric_value.<locals>.wrapper.<locals>.inner_func(*args, **kwargs)
     32 @wraps(metric_fn)
     33 def inner_func(*args, **kwargs):
--> 34     return metric_fn(*args, **kwargs)

File ~/.pyenv/versions/3.9.6/envs/data_pipelines/lib/python3.9/site-packages/great_expectations/expectations/metrics/column_aggregate_metric_provider.py:88, in column_aggregate_value.<locals>.wrapper.<locals>.inner_func(cls, execution_engine, metric_domain_kwargs, metric_value_kwargs, metrics, runtime_configuration)
     86     df = df[df[column_name].notnull()]
--> 88 return metric_fn(
     89     cls,
     90     column=df[column_name],
     91     **metric_value_kwargs,
     92     _metrics=metrics,
     93 )
...
    610     ] = self.resolve_metric_bundle(
    611         metric_fn_bundle=metric_fn_bundle_configurations
    612     )

MetricResolutionError: unhashable type: 'dict'

疑问：这是Bug还是使用方式有误？补充：处理的是JSON文件，怀疑GX暂不支持该场景或选错了Profiler。

原因分析与解决方案 #

这个错误unhashable type: 'dict'核心原因是JSON数据中存在嵌套字典类型的字段，UserConfigurableProfiler默认会遍历所有字段计算统计指标，而字典类型无法被哈希，导致指标计算失败。以下是具体解决方法：

1. 排除嵌套字段 #

初始化UserConfigurableProfiler时，通过exclude_columns参数直接排除包含字典的列，只处理结构化字段：

profiler = UserConfigurableProfiler(
    profile_dataset=validator,
    exclude_columns=["嵌套字段列名1", "嵌套字段列名2"]
)

2. 自定义Profiler规则 #

如果需要保留部分嵌套字段，可通过profiler_config参数自定义规则，只对非嵌套字段生成支持的Expectations：

profiler = UserConfigurableProfiler(
    profile_dataset=validator,
    profiler_config={
        "variables": {},
        "rules": {
            "default": {
                "expectations": [
                    "expect_column_values_to_not_be_null",
                    "expect_column_values_to_be_in_type_list"
                    # 仅保留支持非结构化类型的Expectation，或根据字段类型筛选
                ]
            }
        }
    }
)

3. 预处理JSON数据 #

在导入GX之前，先对JSON数据做扁平化处理，将嵌套字典展开为单个字段，避免字典类型进入Dataset：

# 示例：用pandas扁平化JSON数据
import pandas as pd

df = pd.read_json("your_data.json")
df = pd.json_normalize(df.to_dict('records'))
# 再将处理后的df导入GX作为数据源

额外说明 #

UserConfigurableProfiler对JSON这类半结构化数据的支持确实存在局限性，若场景复杂，建议手动编写针对嵌套字段的自定义Expectations，或选择更适合半结构化数据的自定义Profiler实现。

内容的提问来源于stack exchange，提问作者Imad