如何使用Python去除JSON文件中的重复条目？

可以通过Python遍历并处理每个用户对象的permissions字段，对其中的collections列表进行去重处理，同时保留元素首次出现的顺序（匹配你的期望结果）。

代码实现 #

# 原始数据（已转换为Python列表）
json_data = [
    {
        "authType": "ldap",
        "password": "",
        "permissions": [
            {
                "collections": [
                    "aks9099",
                    "aks9099",
                    "aks9098",
                    "aks9100",
                    "aks9100",
                    "aks9101",
                    "aks9102",
                    "aks9103",
                    "aks9103"
                    
                ],
                "project": "Central Project"
            }
        ],
        "role": "devSecOps",
        "username": "chinq.n@example.com"
    },
    {
        "authType": "ldap",
        "password": "",
        "permissions": [
            {
                "collections": [
                    "aks9099",
                    "aks9098",
                    "aks9098",
                    "aks9100",
                    "aks9101",
                    "aks9102",
                    "aks9102",
                    "aks9103"
                ],
                "project": "Central Project"
            }
        ],
        "role": "devSecOps",
        "username": "chinw.d@example.com"
    },
    {
        "authType": "ldap",
        "password": "",
        "permissions": [
            {
                "collections": [
                    "aks9099",
                    "aks9098",
                    "aks9100",
                    "aks9100",
                    "aks9101",
                    "aks9102",
                    "aks9102",
                    "aks9103"
                ],
                "project": "Central Project"
            }
        ],
        "role": "devSecOps",
        "username": "chins.b@example.com"
    }
]

# 遍历处理每个用户的权限列表
for user in json_data:
    for permission in user["permissions"]:
        # 利用字典键的唯一性去重，Python3.7+保留插入顺序
        permission["collections"] = list(dict.fromkeys(permission["collections"]))

# 查看处理后的结果
print(json_data)

关键逻辑说明 #

• 多层遍历：外层循环遍历所有用户，内层循环遍历每个用户的权限项（支持一个用户有多个权限的场景）。
• 顺序保留去重：dict.fromkeys()会根据列表元素生成无重复的键，且Python3.7及以上版本的字典会严格保留键的插入顺序，完美匹配你期望的元素排列。
• 兼容低版本Python：如果使用Python3.6及以下版本，可替换为手动去重的列表推导式：

seen = set()
permission["collections"] = [x for x in permission["collections"] if not (x in seen or seen.add(x))]

结果验证 #

处理后的json_data将完全符合你提供的期望结果，每个collections列表仅保留唯一元素，且顺序与元素首次出现的顺序一致。

内容的提问来源于stack exchange，提问作者chin wooi Ng