Python列表去重失效疑问：sorted(set(list1))为何未删除重复项？

问题描述 #

我有如下列表：

list1=['philips, be, currently, monitor, development, and, update, relate, to, the, recently, release, oracle, security, update, concern, a, vulnerability, within, the, e, business, suite, product, oracle, have, release, a, patch, and, recommend, that, it, be, apply, as, soon, as, possibleat, this, point, of, time, no, philips, product, be, know, to, be, impact, in, accordance, with, philips, global, security, policy, philips, continue, to, analyze, the, matter, and, further, information, will, be, post, on, the, philips, product, security, advisory, page, as, appropriatephilip, be, commit, to, ensure, the, safety, security, integrity, and, regulatory, compliance, of, our, product, to, be, deploy, and, to, operate, within, philips, approve, product, specification, therefore, in, accordance, with, philips, policy, and, regulatory, requirement, all, change, of, configuration, or, software, to, philips, product, may, be, implement, only, in, accordance, with, philips, product, specific, verify, validate, authorized, and, communicate, customer, procedure, or, field, actionsif, a, product, do, require, operate, system, security, update, configuration, change, or, other, action, to, be, take, by, our, customer, or, by, philips, customer, services, product, specific, service, documentation, will, be, produce, by, philips, product, team, and, make, available, to, philips, service, delivery, platform, such, as, the, philips, incenter, customer, portalcontract, entitle, customer, may, use, philips, incenter, and, be, encourage, to, request, philips, incenter, access, and, reference, product, specific, information, post, if, customer, still, have, question, all, customer, be, encourage, to, contact, their, local, service, support, team, or, regional, product, service, support, as, appropriate, for, up, to, date, information, specific, to, their, philips, product', 'philips, be, currently, monitor, development, and, update, relate, to, the, recently, release, ciscoadvisory, cisco, have, confirm, that, a, critical, vulnerability, exist, in, their, adaptive, security, appliance, software, and, firepower, threat, defense, softwaresuccessful, execution, of, this, vulnerability, could, allow, an, unauthenticated, remote, attacker, to, because, a, denial, of, service, condition, on, an, affected, device, cisco, have, release, software, update, that, help, remediate, this, vulnerabilityat, this, time, no, philips, product, be, know, to, be, impact, in, accordance, with, philips, global, security, policy, philips, continue, to, analyze, the, matter, and, further, information, will, be, post, on, the, philips, product, security, advisory, page, as, appropriatephilip, be, commit, to, ensure, the, safety, security, integrity, and, regulatory, compliance, of, our, product, to, be, deploy, and, to, operate, within, philips, approve, product, specification, therefore, in, accordance, with, philipss, policy, and, regulatory, requirement, all, change, of, configuration, or, software, to, philips, product, may, be, implement, only, in, accordance, with, philipss, product, specific, verify, validate, authorized, and, communicate, customer, procedure, or, field, actionsif, a, product, do, require, operate, system, security, update, configuration, change, or, other, action, to, be, take, by, our, customer, or, by, philips, customer, services, product, specific, service, documentation, will, be, produce, by, philipss, product, team, and, make, available, to, philips, service, delivery, platform, such, as, the, philips, incenter, customer, portalcontract, entitle, customer, may, use, philips, incenter, and, be, encourage, to, request, philips, incenter, access, and, reference, product, specific, information, post, if, customer, still, have, question, all, customer, be, encourage, to, contact, their, local, service, support, team, or, regional, product, service, support, as, appropriate, for, up, to, date, information, specific, to, their, philips, product']

我尝试用以下代码去重：

mylist = sorted(set(list1))
mylist

但结果里原列表的两个元素都被保留了，去没生效，这是为什么？

原因分析 #

去重失败的核心原因是：list1中的两个元素是完全不同的字符串。第一个字符串开头是Oracle安全更新相关内容，包含oracle, security, update等专属片段；第二个字符串开头是Cisco漏洞相关内容，包含ciscoadvisory, cisco等专属片段。

Python的set()去重逻辑是判断整个元素的唯一性——只有当两个元素的内容完全一致时，才会被判定为重复项剔除。这两个字符串虽然后半部分高度相似，但整体是两个独立的对象，因此set()会将它们全部保留。

若需剔除字符串内部的重复单词 #

如果你的需求是清理每个字符串里的重复单词，可以先拆分字符串为单词列表，去重后再重新组合，示例代码如下：

processed_list = []
for s in list1:
    # 拆分字符串为单词列表
    words = s.split(', ')
    # 去重并排序
    unique_words = sorted(set(words))
    # 重新组合为字符串
    processed_str = ', '.join(unique_words)
    processed_list.append(processed_str)

print(processed_list)

内容的提问来源于stack exchange，提问作者xavi