MongoDB副本集因批量删除崩溃：慢查询与ReadConcern问题求助

环境信息 #

• GKE集群，共6节点，每个节点配置16GB共享内存、4核共享CPU
• MongoDB采用Bitnami Helm 13.5.X版本部署，架构为3个工作节点+1个仲裁节点的副本集

问题背景 #

通过port-forwarding连接主节点删除约10万条单条大小2KB的脏数据（此前通过Kubernetes Service转发常连接到从节点），误判资源充足执行操作后，副本集陷入崩溃循环，推测是从节点同步oplog时资源不足引发慢查询。

当前状态 #

Pod状态 #

NAME                                      READY   STATUS                   RESTARTS          AGE
mongodb-0                                 0/1     CrashLoopBackOff         377 (80s ago)     36h
mongodb-1                                 0/1     Running                  1 (134m ago)      27h
mongodb-2                                 0/1     ContainerStatusUnknown   59 (6h ago)       11h
mongodb-arbiter-0                         1/1     Running                  309 (5m40s ago)   2d1h

mongodb-0日志关键信息 #

{"t":{"$date":"2023-07-21T05:38:30.240+00:00"},"s":"I",  "c":"REPL",     "id":21550,   "ctx":"initandlisten","msg":"Replaying stored operations from startPoint (exclusive) to endPoint (inclusive)","attr":{"startPoint":{"$timestamp":{"t":1689641543,"i":5857}},"endPoint":{"$timestamp":{"t":1689641885,"i":1}}}}
{"t":{"$date":"2023-07-21T05:38:30.377+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ReadConcernMajorityNotAvailableYet: Read concern majority reads are currently not possible.","nextWakeupMillis":1200}}
{"t":{"$date":"2023-07-21T05:38:30.415+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"initandlisten","msg":"Slow query","attr":{"type":"command","ns":"local.oplog.rs","command":{"getMore":151664126847388003,"collection":"oplog.rs","$db":"local"},"originatingCommand":{"find":"oplog.rs","filter":{"ts":{"$gte":{"$timestamp":{"t":1689641543,"i":5857}},"$lte":{"$timestamp":{"t":1689641885,"i":1}}}},"readConcern":{},"$db":"local"},"planSummary":"COLLSCAN","cursorid":151664126847388003,"keysExamined":0,"docsExamined":100510,"numYields":100,"nreturned":100509,"queryHash":"23904D31","planCacheKey":"23904D31","reslen":16777108,"locks":{"ParallelBatchWriterMode":{"acquireCount":{"r":29}},"FeatureCompatibilityVersion":{"acquireCount":{"r":123,"w":18}},"ReplicationStateTransition":{"acquireCount":{"w":36}},"Global":{"acquireCount":{"r":123,"w":13,"W":5}},"Database":{"acquireCount":{"r":15,"w":12,"W":1}},"Collection":{"acquireCount":{"r":19,"w":4,"W":4}},"Mutex":{"acquireCount":{"r":34}},"oplog":{"acquireCount":{"w":1}}},"flowControl":{"acquireCount":10,"timeAcquiringMicros":19},"readConcern":{"provenance":"implicitDefault"},"storage":{"data":{"bytesRead":368803082,"timeReadingMicros":5070664},"timeWaitingMicros":{"schemaLock":628}},"protocol":"op_msg","durationMillis":174}}
{"t":{"$date":"2023-07-21T05:38:31.581+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ReadConcernMajorityNotAvailableYet: Read concern majority reads are currently not possible.","nextWakeupMillis":1400}}

日志核心问题：

• 出现ReadConcernMajorityNotAvailableYet错误（因副本集多数节点不可用）
• 对local.oplog.rs的查询为全表扫描（COLLSCAN）的慢查询，同步操作占用大量资源

已尝试操作 #

尝试通过port-forwarding修改oplog大小以缓解同步压力，但因Pod持续崩溃未成功，不确定该方案是否适用。

解决方案建议 #

1. 优先稳定可用节点，减少同步压力 #

• 通过port-forwarding连接处于Running状态的mongodb-1：

  kubectl port-forward mongodb-1 27017:27017
  

• 登录MongoDB shell，将mongodb-1临时设置为单节点实例，脱离原副本集：

  rs.initiate({_id: "your-replica-set-name", members: [{_id: 1, host: "mongodb-1:27017"}]})
  

  注：替换your-replica-set-name为实际副本集名称（可从原Helm配置中获取）

2. 调整MongoDB Pod资源分配 #

修改Helm values配置，提高MongoDB工作节点的资源请求与限制，匹配GKE节点的资源能力：

mongodb:
  resources:
    requests:
      cpu: "2"
      memory: "8Gi"
    limits:
      cpu: "3"
      memory: "12Gi"

执行Helm升级生效：

helm upgrade mongodb bitnami/mongodb -f your-values.yaml

资源调整后，重启崩溃的Pod，让节点有足够资源处理同步操作。

3. 修复崩溃节点的同步问题 #

• 待mongodb-1稳定后，逐个重新加入其他节点：
  先清理mongodb-0和mongodb-2的本地数据（如果数据已损坏），然后通过mongodb-1的shell执行：

  rs.add("mongodb-0:27017")
  rs.add("mongodb-2:27017")
  rs.addArb("mongodb-arbiter-0:27017")
  

• 若节点重新加入后仍崩溃，可直接删除对应Pod的PersistentVolumeClaim（PVC），让节点重新从主节点同步数据：

  kubectl delete pvc data-mongodb-0
  kubectl delete pvc data-mongodb-2
  

  注：此操作会丢失节点本地数据，需确保主节点数据完整。

4. 临时调整ReadConcern参数 #

在MongoDB启动参数中临时添加--setParameter readConcernDefault=local，避免ReadConcernMajorityNotAvailableYet错误影响节点启动。可通过Helm values配置：

mongodb:
  extraFlags:
    - "--setParameter readConcernDefault=local"

集群稳定后，再移除该参数恢复默认配置。

5. 调整oplog大小（集群稳定后操作） #

通过Helm配置直接设置oplog大小，无需手动修改：

mongodb:
  oplogSizeGB: 20

执行Helm升级后，MongoDB会自动调整oplog大小，避免因oplog过小导致同步时扫描大量历史数据。

6. 后续数据操作优化 #

• 批量删除数据时，采用分批删除（每次删除1000条），避免一次性生成大量oplog导致同步压力剧增：

  while (db.your-collection.deleteMany({/* 删除条件 */}, {limit: 1000}).deletedCount > 0) {}
  

• 优先通过主节点执行写操作，避免从节点同步压力异常。

内容的提问来源于stack exchange，提问作者Zaki