MongoDB主节点仍尝试连接已移除仲裁节点问题求助

背景 #

生产环境MongoDB副本集出现复制故障，我们删除了主节点和仲裁节点的local数据库，仅用主节点重新初始化副本集以保障应用临时运行。但现在出现新问题：尽管rs.status()显示副本集只剩主节点，主节点仍持续尝试连接已停止的仲裁节点，引发Too many open files错误导致数据库宕机。已临时提升文件打开限制，但问题未解决，需要在重新搭建完整副本集前找到临时解决方案。

相关日志信息 #

2023-07-10T02:04:55.287+0300 I CONNPOOL [Replication] Dropping all pooled connections to ArbiterID:27017 due to HostUnreac
hable: Error connecting to 10.202.2.48:27017 :: caused by :: Too many open files
2023-07-10T02:04:55.287+0300 E -        [Replication] cannot open /dev/urandom Too many open files
2023-07-10T02:04:55.287+0300 F -        [Replication] Fatal Assertion 28839 at src/mongo/platform/random.cpp 161
2023-07-10T13:01:09.802+0300 I REPL_HB  [replexec-222] Error in heartbeat (requestId: 16678428) to ArbiterID:27017, response status: HostUnreachable: Error connecting to ArbiterID:27017 :: caused by :: Connection refused
2023-07-10T13:01:09.802+0300 I ASIO     [Replication] Connecting to ArbiterID:27017
2023-07-10T13:01:09.802+0300 I ASIO     [Replication] Failed to connect to ArbiterID:27017 - HostUnreachable: Error connecting to ArbiterID:27017 :: caused by :: Connection refused
2023-07-10T13:01:09.802+0300 I CONNPOOL [Replication] Dropping all pooled connections to ArbiterID:27017 due to HostUnreachable: Error connecting to ArbiterID:27017 :: caused by :: Connection refused

rs.status()输出结果 #

{
        "set" : "UHI-PROD",
        "date" : ISODate("2023-07-10T10:02:35.295Z"),
        "myState" : 1,
        "term" : NumberLong(4),
        "syncingTo" : "",
        "syncSourceHost" : "",
        "syncSourceId" : -1,
        "heartbeatIntervalMillis" : NumberLong(2000),
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1688983352, 4742),
                        "t" : NumberLong(4)
                },
                "readConcernMajorityOpTime" : {
                        "ts" : Timestamp(1688983352, 4742),
                        "t" : NumberLong(4)
                },
                "appliedOpTime" : {
                        "ts" : Timestamp(1688983352, 4742),
                        "t" : NumberLong(4)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1688983352, 4742),
                        "t" : NumberLong(4)
                }
        },
        "lastStableCheckpointTimestamp" : Timestamp(1688983291, 2),
        "electionCandidateMetrics" : {
                "lastElectionReason" : "electionTimeout",
                "lastElectionDate" : ISODate("2023-07-10T07:45:09.406Z"),
                "electionTerm" : NumberLong(4),
                "lastCommittedOpTimeAtElection" : {
                        "ts" : Timestamp(0, 0),
                        "t" : NumberLong(-1)
                },
                "lastSeenOpTimeAtElection" : {
                        "ts" : Timestamp(1688943890, 8),
                        "t" : NumberLong(3)
                },
                "numVotesNeeded" : 1,
                "priorityAtElection" : 1,
                "electionTimeoutMillis" : NumberLong(10000),
                "newTermStartDate" : ISODate("2023-07-10T07:45:09.413Z"),
                "wMajorityWriteAvailabilityDate" : ISODate("2023-07-10T07:45:09.500Z")
        },
        "members" : [
                {
                        "_id" : 0,
                        "name" : "***:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 8249,
                        "optime" : {
                                "ts" : Timestamp(1688983352, 4742),
                                "t" : NumberLong(4)
                        },
                        "optimeDate" : ISODate("2023-07-10T10:02:32Z"),
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "electionTime" : Timestamp(1688975109, 1),
                        "electionDate" : ISODate("2023-07-10T07:45:09Z"),
                        "configVersion" : 2,
                        "self" : true,
                        "lastHeartbeatMessage" : ""
                }
        ],
        "ok" : 1,
        "operationTime" : Timestamp(1688983352, 4742),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1688983352, 4742),
                "signature" : {
                        "hash" : BinData(0,"s7Diwi3gtsej746SjJOzhIkdp/E="),
                        "keyId" : NumberLong("7195139362714025986")
                }
        }
}

临时解决方法 #

• 重启主节点MongoDB服务：安排维护窗口，重启主节点后，内存中残留的旧仲裁节点信息会被清除，仅加载local库中当前的单节点副本集配置。注意提前协调应用侧做临时降级或切换，避免影响业务。
• 强制刷新副本集配置：在主节点的mongo shell中执行以下命令，强制重新加载当前配置，覆盖内存中的旧成员信息：

  rs.reconfig(rs.config(), {force: true})
  

  单节点副本集下使用force参数安全有效，确保配置中仅包含当前主节点。
• 临时调大心跳间隔：若重启不可行，可临时调大心跳间隔，减少连接尝试频率，缓解文件句柄消耗：

  const newConfig = rs.config();
  newConfig.heartbeatIntervalMillis = 30000; // 改为30秒一次
  rs.reconfig(newConfig, {force: true})
  

• 确认系统级文件限制生效：检查操作系统的ulimit设置（执行ulimit -n），同时确认MongoDB配置文件中的maxOpenFiles参数已调整，调整后需重启MongoDB才能生效。

内容的提问来源于stack exchange，提问作者Alaa Mohsen