MongoDB独立服务器随机崩溃（Invariant failure）排查求助

生产环境使用Docker部署单节点MongoDB（无副本集），通过PHP和Python代码访问。服务器会随机重启，抛出Invariant failure错误，无法定位根因。已在MongoDB开发者社区发帖、提交Jira工单，但未获回复。

崩溃核心日志 #

{"t":{"$date":"2023-09-06T06:30:50.517+00:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"conn3","msg":"WiredTiger error message","attr":{"error":22,"message":{"ts_sec":1693981850,"ts_usec":517620,"thread":"1:0x7f9c42ef9640","session_name":"WT_SESSION.get_key","category":"WT_VERB_DEFAULT","category_id":9,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__wt_txn_context_prepare_check:19:not permitted in a prepared transaction","error_str":"Invalid argument","error_code":22}}}
{"t":{"$date":"2023-09-06T06:30:50.517+00:00"},"s":"F",  "c":"ASSERT",   "id":23083,   "ctx":"conn3","msg":"Invariant failure","attr":{"expr":"cursor->get_key(cursor, &recordId)","error":"BadValue: 22: Invalid argument","file":"src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp","line":2552}}
{"t":{"$date":"2023-09-06T06:30:50.517+00:00"},"s":"F",  "c":"ASSERT",   "id":23084,   "ctx":"conn3","msg":"***aborting after invariant() failure"}}
{"t":{"$date":"2023-09-06T06:30:50.517+00:00"},"s":"F",  "c":"CONTROL",  "id":6384300, "ctx":"conn3","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted)."}}

完整崩溃日志已归档。

环境信息与已尝试的排查操作 #

• MongoDB版本：6.0.9，featureCompatibilityVersion设置为6.0；曾尝试升级至7.0版本，无效
• 已执行的排查动作：
  • 导出数据迁移至6.0、6.0.9、7.0版本的MongoDB实例，崩溃仍发生
  • 更新PHP和Python的MongoDB驱动至最新版本，无效
  • 执行mongod --repair命令修复数据库，结果显示无异常
  • 执行mongod --upgrade命令升级数据库，无效果，崩溃依旧

服务器随机重启会导致生产服务停滞10秒以上，目前无明确排查方向。

更新1：高日志级别下的慢查询线索 #

提高MongoDB日志级别至3后，发现相关日志：

{"t":{"$date":"2023-09-06T07:24:05.844+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn1023","msg":"Slow query","attr":{"type":"command","ns":"vpn.session","command":{"find":"session","filter":{"access_token":"<hidden>"},"limit":1,"$db":"vpn","lsid":{"id":{"$uuid":"a44079c4-ebd9-4126-8aaf-bf439aecda11"}}}},"planSummary":"COLLSCAN","keysExamined":0,"docsExamined":145708,"cursorExhausted":true,"numYields":145,"nreturned":1,"queryHash":"CAEA5ECC","planCacheKey":"CAEA5ECC","queryFramework":"classic","reslen":384,"locks":{"FeatureCompatibilityVersion":{"acquireCount":{"r":146}},"Global":{"acquireCount":{"r":146}},"Mutex":{"acquireCount":{"r":1}}},"storage":{},"remote":"172.18.0.1:57968","protocol":"op_msg","durationMillis":30}}
{"t":{"$date":"2023-09-06T07:24:05.844+00:00"},"s":"D2", "c":"QUERY",    "id":22783,   "ctx":"conn1023","msg":"Received interrupt request for unknown op","attr":{"opId":296291,"knownOps":[]}}
{"t":{"$date":"2023-09-06T07:24:05.844+00:00"},"s":"D3", "c":"-",        "id":5127803, "ctx":"conn1023","msg":"Released the Client","attr":{"client":"conn1023"}}
{"t":{"$date":"2023-09-06T07:24:05.844+00:00"},"s":"D3", "c":"-",        "id":5127801, "ctx":"conn1023","msg":"Setting the Client","attr":{"client":"conn1023"}}

近期将PHP开发的Wireguard Manager从MySQL迁移至MongoDB时，实习生未给vpn.session集合创建索引。该集合有数百万文档，每次查询access_token都会触发全表扫描（COLLSCAN）的慢查询，同时日志显示连接存在“Received interrupt request for unknown op”的无效操作。怀疑无索引导致的慢查询引发竞争条件，进而触发Invariant failure断言错误导致服务器崩溃，但不确定具体关联机制。

已计划为vpn.session集合创建access_token字段的索引，观察是否停止崩溃，但仍需明确根因及彻底解决方案。

内容的提问来源于stack exchange，提问作者Sibidharan