K8s中MongoDB副本集恢复后mongodb-1出现CrashLoopBackOff问题咨询

在Kubernetes集群中部署了包含2个MongoDB副本节点（mongodb-0、mongodb-1）和1个仲裁节点（mongodb-arbiter-0）的副本集，每个副本节点均启用了mongodb-exporter。此前从AWS EC2卷快照恢复数据库的操作可正常执行，但本次恢复后，mongodb-0与仲裁节点能正常启动，mongodb-1却出现CrashLoopBackOff状态。查看日志发现，mongodb-metrics-exporter报错无法连接到localhost:27017，还触发了空指针引用的运行时panic。

Pod状态信息 #

my-feature            mongodb-0                                                         2/2     Running   0               41h
my-feature            mongodb-1                                                         2/2     Running   0               41h
my-feature            mongodb-arbiter-0                                                 1/1     Running   0               41h

错误日志 #

Stream closed EOF for my-feature/mongodb-1 (mongodb)
metrics time="2023-08-28T05:23:15Z" level=error msg="Cannot connect to MongoDB: cannot connect to MongoDB: server selection error: context canceled, current topology: { Type: Single, Servers: [{ Addr: localhost:27017, Type: Unknown, Last error: connection() error occurred during connection handshake: dial tcp [::1]:27017: connect: connection refused }, ] }"
metrics 2023/08/28 05:23:15 http: panic serving 10.0.2.198:44132: runtime error: invalid memory address or nil pointer dereference
metrics goroutine 97 [running]:
metrics net/http.(*conn).serve.func1()
metrics     /opt/hostedtoolcache/go/1.17.11/x64/src/net/http/server.go:1802 +0xb9
metrics panic({0xb36880, 0x1264f80})
metrics     /opt/hostedtoolcache/go/1.17.11/x64/src/runtime/panic.go:1047 +0x266
metrics go.mongodb.org/mongo-driver/mongo.newDatabase(0x0, {0xbf9d4a, 0x5}, {0x0, 0x30, 0xc00029e400})
metrics     /home/runner/go/pkg/mod/go.mongodb.org/mongo-driver@v1.9.1/mongo/database.go:47 +0x5c
metrics go.mongodb.org/mongo-driver/mongo.(*Client).Database(...)
metrics     /home/runner/go/pkg/mod/go.mongodb.org/mongo-driver@v1.9.1/mongo/client.go:837
metrics github.com/percona/mongodb_exporter/exporter.getClusterRole({0xd97d80, 0xc00009a660}, 0x40d1f4)
metrics     /home/runner/work/mongodb_exporter/mongodb_exporter/exporter/topology_info.go:170 +0x8b
metrics github.com/percona/mongodb_exporter/exporter.(*topologyInfo).loadLabels(0xc000099560, {0xd97d80, 0xc00009a660})
metrics     /home/runner/work/mongodb_exporter/mongodb_exporter/exporter/topology_info.go:103 +0xeb
metrics github.com/percona/mongodb_exporter/exporter.newTopologyInfo({0xd97d80, 0xc00009a660}, 0x0)
metrics     /home/runner/work/mongodb_exporter/mongodb_exporter/exporter/topology_info.go:73 +0x8b
metrics github.com/percona/mongodb_exporter/exporter.(*Exporter).Handler.func1({0xd95020, 0xc00048a000}, 0xc000486000)
metrics     /home/runner/work/mongodb_exporter/mongodb_exporter/exporter/exporter.go:304 +0x3a5
metrics net/http.HandlerFunc.ServeHTTP(0x0, {0xd95020, 0xc00048a000}, 0x0)
metrics     /opt/hostedtoolcache/go/1.17.11/x64/src/net/http/server.go:2047 +0x2f
metrics net/http.(*ServeMux).ServeHTTP(0x0, {0xd95020, 0xc00048a000}, 0xc000486000)
metrics     /opt/hostedtoolcache/go/1.17.11/x64/src/net/http/server.go:2425 +0x149
metrics net/http.serverHandler.ServeHTTP({0xc000098b10}, {0xd95020, 0xc00048a000}, 0xc000486000)
metrics     /opt/hostedtoolcache/go/1.17.11/x64/src/net/http/server.go:2879 +0x43b
metrics net/http.(*conn).serve(0xc0000ba1e0, {0xd97db8, 0xc000346c60})
metrics     /opt/hostedtoolcache/go/1.17.11/x64/src/net/http/server.go:1930 +0xb08
metrics created by net/http.(*Server).Serve
metrics     /opt/hostedtoolcache/go/1.17.11/x64/src/net/http/server.go:3034 +0x4e8

解决建议 #

• 排查MongoDB主进程状态：虽然Pod显示2/2 Running，但日志里存在Stream closed EOF for my-feature/mongodb-1 (mongodb)，说明mongodb容器进程可能异常退出。进入mongodb-1的mongodb容器，执行ps aux | grep mongod确认进程是否存活；若进程不存在，查看mongodb容器的启动日志，排查数据恢复后的初始化错误，比如权限问题、数据目录损坏、副本集配置不匹配。
• 检查exporter连接配置：确认mongodb-exporter的连接参数是否正确。如果exporter配置连接localhost:27017，但mongodb进程未启动或监听地址不是localhost（比如绑定集群内部IP），会导致连接失败。可修改exporter的连接地址为mongodb容器内部IP，同时确保启动命令包含正确的认证信息（若MongoDB开启认证）。
• 升级mongodb-exporter版本：日志显示该panic来自旧版本的percona mongodb-exporter（基于v1.9.1的mongo-driver），旧版本在连接失败后的处理逻辑存在缺陷。升级到最新稳定版可修复这类空指针异常问题。
• 验证副本集状态：在mongodb-0上执行rs.status()查看副本集成员状态，确认mongodb-1是否被正确识别为副本节点。若恢复后副本集配置异常，mongodb-1可能无法加入集群导致进程启动失败，需手动重新初始化副本集或调整成员配置。
• 检查存储卷状态：查看mongodb-1挂载的EC2快照恢复卷是否正常，执行kubectl describe pv <mongodb-1-pv-name>检查卷的挂载状态、权限配置（比如容器用户是否有数据目录读写权限）。可尝试卸载后重新挂载存储卷，或校验快照恢复的数据完整性。

内容的提问来源于stack exchange，提问作者PikaLong