AWS Glue ETL Job数据处理：空值转null并清理特殊类型键

原代码问题分析 #

你的函数存在两个核心问题：

1. 第一个return new_dict语句直接终止了函数执行，后面的键类型处理逻辑完全不会运行
2. 两段逻辑重复冗余，没有合并成统一的递归处理流程

正确实现代码 #

下面是合并了两项需求的递归函数，可直接用于AWS Glue ETL作业：

import json

def clean_data_recursive(rec):
    # 处理字典类型
    if isinstance(rec, dict):
        new_dict = {}
        for key, value in rec.items():
            # 需求2：如果键是类型对象（如int、str这类type实例），直接移除该键
            if isinstance(key, type):
                continue
            
            # 递归处理值
            processed_value = clean_data_recursive(value)
            
            # 过滤空字典/空列表，避免无效数据残留
            if processed_value not in [{}, []]:
                new_dict[key] = processed_value
        return new_dict
    
    # 处理列表类型
    elif isinstance(rec, list):
        new_list = []
        for item in rec:
            processed_item = clean_data_recursive(item)
            # 空值转"null"，非空项加入列表
            if processed_item in ["", None]:
                new_list.append("null")
            else:
                new_list.append(processed_item)
        return new_list
    
    # 处理基础数据类型
    else:
        # 需求1：空字符串或None转换为"null"
        if rec in ["", None]:
            return "null"
        # 其他值直接返回
        else:
            return rec

# 测试代码
json_data = '''
{
    "ptype#id": "USER#748f610a488a9327b9609b151a0fb3b2e4f23e8925fe89353d5fe8f5ce89d8ea",
    "stype#sk": "USER#748f610a488a9327b9609b151a0fb3b2e4f23e8925fe89353d5fe8f5ce89d8ea",
    "accountCreatedDate": "",
    "accountStatus": "",
    "anonymUserID": "4be17719-df5e-43e3-b2fe-c10c0a858b4c",
    "app": "OCM",
    "appOrigin": "OCM",
    "emailAddress": "test2.jnelson@yopmail.com",
    "migration_time": 1679390468329,
    "ofs": {
        "accountStatus": "active",
        "attributes": {
            "accountType": "saving",
            "appSettings": {
                "string": "",
                "struct": ""
            },
            "ccPolicy": {
                "array": "",
                "string": ""
            },
            "country": "US",
            "goals": {
                "string": "",
                "struct": ""
            },
            "personalPref": {
                "string": "",
                "struct": ""
            },
            "phoneLocale": "en-us",
            "policy": {
                "df": [
                    {
                        "v1": 1,
                        "v2": "",
                        "v3": "",
                        "v4": "",
                        "v5": ""
                    }
                ],
                "string": ""
            },
            "subscriptionOC": {
                "autoRenewStatus": "",
                "pendingRenewalInfo": {
                    "expiryTimeMillis": "1672444800000",
                    "productId": "com.omronhealthcare.omronconnect.premium.monthly"
                },
                "subscriptionEndDate": "1672444800000",
                "subscriptionPackage": "com.omronhealthcare.omronconnect.premium.monthly",
                "subscriptionStartDate": 1650000000000,
                "subscriptionStatus": "subscribed",
                "subscriptionTrackPackage": [
                    {
                        "isManual": "1",
                        "packageName": "com.omronhealthcare.omronconnect.premium.monthly",
                        "subscribedDate": 1650000000000,
                        "subscriptionStartDate": 1650000000000
                    }
                ],
                "userLoginDate": "hai"
            },
            "weightRange": ""
        },
        "cloudOpt": {},
        "emailAddress": "test2.jnelson@yopmail.com",
        "optIn": {
            "boolean": false
        },
        "walgreens": "hai mere pass"
    },
    "weightInsightRandomID": "hjgeg",
    "weightInsightWeeklyID": ""
}
'''

b = json.loads(json_data)
clean_dict = clean_data_recursive(b)
print(json.dumps(clean_dict, indent=2))

实现说明 #

• 空值转换：所有空字符串""和None都会被统一转换为字符串"null"，支持嵌套字典、列表中的深度处理
• 类型键移除：如果字典的键是type类型的实例（比如int、str这类类型对象），会直接跳过该键值对，不保留在结果中
• 冗余数据过滤：自动过滤处理后产生的空字典和空列表，避免无效数据残留
• Glue环境兼容：代码完全适配AWS Glue的Python运行环境，可直接嵌入ETL作业中使用

内容的提问来源于stack exchange，提问作者Ankit Tiwari