Pymongo批量Upsert报E11000重复键错误及相关疑问

环境信息 #

• MongoDB版本：3.4.24
• PyMongo版本：3.13.0

错误日志 #

{'writeErrors': [
{'index': 0, 
 'code': 11000, 
 'errmsg': "E11000 duplicate key error collection: nap_poc.accesslink index: _id_ dup key: { : ObjectId('6442995bf9606c000d3ae2a9') }", 
 'op': SON([('q', {'a_link_id': 'GLWBA1-EBA-A1_1/3/3', '_id': {'$in': [ObjectId('6442995bf9606c000d3ae2a9')]}}), 
            ('u', {'service_saps': [{'_id': '1/3/3:101', 'qos_policy_in': '825', 'qos_policy_out': '1', 'ip_filter_in': '9020', 'ip_filter_out': '', 'mac_filter_in': '', 'mac_filter_out': '', 'description': 'vf=4445:cn=VODAMS:tl=3C03333108:st=act:', 'parameters': {'customer_id': '1714', 'customer_name': '1714', 'customer_description': 'Vodafone UK Mobile Services'}, 'service': {'_id': '2111000', 'name': 'VF_RAN_SPK_OMCH_VRF1', 'type': 'vprn', 'number_of_saps': 1, 'service_category': '', 'parameters': {'admin_state': 'enabled', 'vrf_import': ['VF_RAN_OMCH_VRF_IMPORT'], 'vrf_export': ['VF_RAN_OMCH_VRF_EXPORT'], 'route_distinguisher': '4445:211537', 'autonomous_system': '4445', 'maximum_routes': '2000', 'maximum_routes_threshold': '80'}, 'description': 'VF_RAN_OMCH'}, 'related_service': {}, 'related_service_sap': {'_id': '', 'name': '', 'description': '', 'classification_tag': 'unclassified', 'status': 'Discovered'}, 'service_interface': {'name': 'VODAMS_007920_920_01', 'type': 'vprn', 'vlan': '101', 'ip_v4_addresses': ['100.123.146.5/30'], 'ip_v6_addresses': [], 'static_routes': [], 'static_routes_ipv6': [], 'bgp_neighbors': [{'neighbor': '100.123.146.6', 'family': 'ipv4', 'description': 'VODAMS_007920_920_01', 'prefix_limit_ipv4': '2000', 'prefix_limit_threshold_ipv4': '80', 'bfd_enabled': 'enabled', 'parameters': {'prefix_limit': {'ipv4': {'limit': '2000', 'threshold': '80', 'log_only': 'log-only'}}}, 'group': {'peer_AS_number': '65068', 'export_policy': ['VF_RAN_OMCH_BGP_EXPORT_TO_CSG'], 'auth_key': 'qJISkqx+/G6JseefC11oDLhtLveJDSR5A4Cfb59Cdw==', 'name': 'VF_RAN_SPK_OMCH'}}], 'bgp_ipv6_neighbors': [], 'ospf_neighbors': [], 'parameters': {'bfd_ipv4': {'tx_interval': '150', 'receive': '150', 'multiplier': '3'}, 'unmatched_lines': '                icmp\n                    no mask-reply\n                    no redirects\n                exit'}, 'description': 'vf=4445:cn=VODAMS:tl=3C03333108:st=act:', 'ip_mtu': '1992'}, 'classification_tag': 'unclassified', 'status': 'Discovered', 'mtu': '', 'name': '1/3/3:101'}, 
                                    {'_id': '1/3/3:201', 'qos_policy_in': '825', 'qos_policy_out': '1', 'ip_filter_in': '9020', 'ip_filter_out': '', 'mac_filter_in': '', 'mac_filter_out': '', 'description': 'vf=4445:cn=VODAMS:tl=3C03333120:st=act:', 'parameters': {'customer_id': '1714', 'customer_name': '1714', 'customer_description': 'Vodafone UK Mobile Services'}, 'service': {'_id': '2121000', 'name': 'VF_RAN_SPK_UNENCRYPT_VRF1', 'type': 'vprn', 'number_of_saps': 1, 'service_category': '', 'parameters': {'admin_state': 'enabled', 'vrf_import': ['VF_RAN_UNENCRYPT_VRF_IMPORT'], 'vrf_export': ['VF_RAN_UNENCRYPT_VRF_EXPORT'], 'route_distinguisher': '4445:212537', 'autonomous_system': '4445', 'maximum_routes': '2000', 'maximum_routes_threshold': '80'}, 'description': 'VF_RAN_UNENCRYPT'}, 'related_service': {}, 'related_service_sap': {'_id': '', 'name': '', 'description': '', 'classification_tag': 'unclassified', 'status': 'Discovered'}, 'service_interface': {'name': 'VODAMS_007920_920_01', 'type': 'vprn', 'vlan': '201', 'ip_v4_addresses': ['100.124.146.5/30'], 'ip_v6_addresses': [], 'static_routes': [], 'static_routes_ipv6': [], 'bgp_neighbors': [{'neighbor': '100.124.146.6', 'family': 'ipv4', 'description': 'VODAMS_007920_920_01', 'prefix_limit_ipv4': '2000', 'prefix_limit_threshold_ipv4': '80', 'policies_out': ['VF_RAN_UNENCRYPT_BGP_EXPORT_TO_VODAMS_007920_920_01'], 'bfd_enabled': 'enabled', 'parameters': {'prefix_limit': {'ipv4': {'limit': '2000', 'threshold': '80', 'log_only': 'log-only'}}}, 'group': {'peer_AS_number': '65068', 'auth_key': 'qJISkqx+/G6JseefC11oDJJlW3xS2emOa+8Xm/O2eg==', 'name': 'VF_RAN_SPK_UNENCRYPT'}}], 'bgp_ipv6_neighbors': [], 'ospf_neighbors': [], 'parameters': {'bfd_ipv4': {'tx_interval': '150', 'receive': '150', 'multiplier': '3'}, 'unmatched_lines': '                icmp\n                    no mask-reply\n                    no redirects\n                exit'}, 'description': 'vf=4445:cn=VODAMS:tl=3C03333120:st=act:', 'ip_mtu': '1992'}, 'classification_tag': 'unclassified', 'status': 'Discovered', 'mtu': '', 'name': '1/3/3:201'}, 
                                    {'_id': '1/3/3:501', 'qos_policy_in': '825', 'qos_policy_out': '1', 'ip_filter_in': '9020', 'ip_filter_out': '', 'mac_filter_in': '', 'mac_filter_out': '', 'description': 'vf=4445:cn=VODAMS:tl=3C03333126:st=act:', 'parameters': {'customer_id': '1714', 'customer_name': '1714', 'customer_description': 'Vodafone UK Mobile Services'}, 'service': {'_id': '2131000', 'name': 'VF_RAN_SPK_ENCRYPT_VRF1', 'type': 'vprn', 'number_of_saps': 1, 'service_category': '', 'parameters': {'admin_state': 'enabled', 'vrf_import': ['VF_RAN_ENCRYPT_VRF_IMPORT'], 'vrf_export': ['VF_RAN_ENCRYPT_VRF_EXPORT'], 'route_distinguisher': '4445:213537', 'autonomous_system': '4445', 'maximum_routes': '2000', 'maximum_routes_threshold': '80'}, 'description': 'VF_RAN_ENCRYPT'}, 'related_service': {}, 'related_service_sap': {'_id': '', 'name': '', 'description': '', 'classification_tag': 'unclassified', 'status': 'Discovered'}, 'service_interface': {'name': 'VODAMS_007920_920_01', 'type': 'vprn', 'vlan': '501', 'ip_v4_addresses': ['100.125.146.5/30'], 'ip_v6_addresses': [], 'static_routes': [], 'static_routes_ipv6': [], 'bgp_neighbors': [{'neighbor': '100.125.146.6', 'family': 'ipv4', 'description': 'VODAMS_007920_920_01', 'prefix_limit_ipv4': '2000', 'prefix_limit_threshold_ipv4': '80', 'bfd_enabled': 'enabled', 'parameters': {'prefix_limit': {'ipv4': {'limit': '2000', 'threshold': '80', 'log_only': 'log-only'}}}, 'group': {'peer_AS_number': '65068', 'export_policy': ['VF_RAN_ENCRYPT_BGP_EXPORT_TO_CSG'], 'auth_key': 'qJISkqx+/G6JseefC11oDIQDsIcx6guLewjBB/n6HA==', 'name': 'VF_RAN_SPK_ENCRYPT'}}], 'bgp_ipv6_neighbors': [], 'ospf_neighbors': [], 'parameters': {'bfd_ipv4': {'tx_interval': '150', 'receive': '150', 'multiplier': '3'}, 'unmatched_lines': '                icmp\n                    no mask-reply\n                    no redirects\n                exit'}, 'description': 'vf=4445:cn=VODAMS:tl=3C03333126:st=act:', 'ip_mtu': '1992'}, 'classification_tag': 'unclassified', 'status': 'Discovered', 'mtu': '', 'name': '1/3/3:501'}, 
                                    {'_id': '1/3/3:601', 'qos_policy_in': '825', 'qos_policy_out': '1', 'ip_filter_in': '9020', 'ip_filter_out': '', 'mac_filter_in': '', 'mac_filter_out': '', 'description': 'vf=4445:cn=VODAMS:tl=3C03333136:st=act:', 'parameters': {'customer_id': '1714', 'customer_name': '1714', 'customer_description': 'Vodafone UK Mobile Services'}, 'service': {'_id': '2141000', 'name': 'VF_RAN_SPK_TEF_VRF1', 'type': 'vprn', 'number_of_saps': 1, 'service_category': '', 'parameters': {'admin_state': 'enabled', 'vrf_import': ['VF_RAN_TEF_VRF_IMPORT'], 'vrf_export': ['VF_RAN_TEF_VRF_EXPORT'], 'route_distinguisher': '4445:214537', 'autonomous_system': '4445', 'maximum_routes': '2000', 'maximum_routes_threshold': '80'}, 'description': 'VF_RAN_TEF'}, 'related_service': {}, 'related_service_sap': {'_id': '', 'name': '', 'description': '', 'classification_tag': 'unclassified', 'status': 'Discovered'}, 'service_interface': {'name': 'VODAMS_007920_920_01', 'type': 'vprn', 'vlan': '601', 'ip_v4_addresses': ['100.126.146.5/30'], 'ip_v6_addresses': [], 'static_routes': [], 'static_routes_ipv6': [], 'bgp_neighbors': [{'neighbor': '100.126.146.6', 'family': 'ipv4', 'description': 'VODAMS_007920_920_01', 'prefix_limit_ipv4': '2000', 'prefix_limit_threshold_ipv4': '80', 'bfd_enabled': 'enabled', 'parameters': {'prefix_limit': {'ipv4': {'limit': '2000', 'threshold': '80', 'log_only': 'log-only'}}}, 'group': {'peer_AS_number': '65068', 'export_policy': ['VF_RAN_TEF_BGP_EXPORT_TO_CSG'], 'auth_key': 'qJISkqx+/G6JseefC11oDHRAesB7207OQRLLcm8nNQ==', 'name': 'VF_RAN_SPK_TEF'}}], 'bgp_ipv6_neighbors': [], 'ospf_neighbors': [], 'parameters': {'bfd_ipv4': {'tx_interval': '150', 'receive': '150', 'multiplier': '3'}, 'unmatched_lines': '                icmp\n                    no mask-reply\n                    no redirects\n                exit'}, 'description': 'vf=4445:cn=VODAMS:tl=3C03333136:st=act:', 'ip_mtu': '1992'}, 'classification_tag': 'unclassified', 'status': 'Discovered', 'mtu': '', 'name': '1/3/3:601'}], 
                   'a_link_id': 'GLWBA1-EBA-A1_1/3/3', 
                   'a_link_name': 'GLWBA1-EBA-A1_1/3/3'}), 
            ('multi', False), 
            ('upsert', True)])}, 
...
],
'writeConcernErrors': [], 
'nInserted': 0, 
'nUpserted': 0, 
'nMatched': 0, 
'nModified': 0, 
'nRemoved': 0, 
'upserted': []}

业务说明 #

• access_link集合中access_link_id字段无索引或唯一约束，允许多个文档拥有相同的access_link_id
• 当前Upsert逻辑：匹配a_link_id和ObjectId则更新文档，否则生成新ObjectId插入新文档

疑问 #

1. 为何此次Upsert被判定为插入操作，且插入的ObjectId与现有ObjectId冲突？
2. MongoDB ObjectId是如何生成的？
3. 通过mongodump和mongorestore从其他服务器复制数据库后，新生成的ObjectId会受什么影响？

解答 #

1. Upsert触发插入且_id冲突的原因 #

从错误日志的查询条件q可以看到，当前查询用了{'_id': {'$in': [ObjectId('6442995bf9606c000d3ae2a9')]}}，MongoDB的Upsert逻辑是：当查询条件未匹配到任何文档时，会基于查询条件里的字段生成新文档。这里查询条件包含了_id，所以插入时会直接使用这个_id值，而非生成新的ObjectId，刚好这个_id已存在于集合中，就触发了唯一键冲突。

查询未匹配到的核心原因是：集合中存在_id=6442995bf9606c000d3ae2a9的文档，但该文档的a_link_id不是GLWBA1-EBA-A1_1/3/3，导致a_link_id+_id的组合条件不匹配，Upsert就尝试插入带该_id的新文档，最终引发冲突。

2. MongoDB ObjectId的生成规则 #

MongoDB的ObjectId是12字节的二进制值，由四部分组成：

• 前4字节：生成时的时间戳（精确到秒）
• 接下来3字节：生成该ObjectId的MongoDB实例所在的机器标识符
• 再接下来2字节：生成该ObjectId的进程ID（PID）
• 最后3字节：自增计数器（同一秒内、同一机器、同一进程生成的ObjectId，计数器会递增）

这种结构确保了分布式环境下，不同实例生成的ObjectId几乎不会重复。

3. mongodump/mongorestore对新生成ObjectId的影响 #

mongodump和mongorestore仅复制现有文档的_id，不会修改MongoDB实例的机器