GKE集群中ETCD节点每日凌晨自动退出集群问题求助

环境信息 #

• 3节点GKE集群，运行多个业务应用
• 部署APISIX网关：chart为apisix，repoURL为https://charts.apiseven.com，targetRevision为0.11.0
• APISIX同步部署3节点etcd集群，版本为3.4.14
• 此前该GKE集群上的Vault集群也出现过类似节点被移除的异常

故障现象 #

集群初始化后运行正常，但每日凌晨5点，第三个etcd节点会退出集群，第二个节点状态保持正常，该现象已多次出现。

相关节点日志 #

etcd-0 节点日志 #

2022-09-29 04:59:04.652 CEST etcd {"caller":"etcdserver/zap_raft.go:77", "level":"info", "logger":"raft", "msg":"90126cc714381e07 switched to configuration voters=(3177002992052145560 10381479693335928327)", "ts":"2022-09-29T02:59:04.652Z"}
2022-09-29 04:59:04.653 CEST etcd {"caller":"membership/cluster.go:472", "cluster-id":"b0d7015fda1525c8", "level":"info", "local-member-id":"90126cc714381e07", "msg":"removed member", "removed-remote-peer-id":"3ff1b5cd453a87df", "removed-remote-peer-urls":[…], "ts":"2022-09-29T02:59:04.653Z"}
2022-09-29 04:59:04.653 CEST etcd {"caller":"rafthttp/peer.go:330", "level":"info", "msg":"stopping remote peer", "remote-peer-id":"3ff1b5cd453a87df", "ts":"2022-09-29T02:59:04.653Z"}

etcd-1 节点日志 #

04:59:04.655 CEST{caller: rafthttp/stream.go:421, error: EOF, level: warn, local-member-id: 3ff1b5cd453a87df, msg: lost TCP streaming connection with remote peer, remote-peer-id: 90126cc714381e07, stream-reader-type: stream MsgApp v2, ts: 2022-09-29T02:59:04.654Z}
04:59:04.678 CEST{caller: rafthttp/stream.go:421, error: EOF, level: warn, local-member-id: 3ff1b5cd453a87df, msg: lost TCP streaming connection with remote peer, remote-peer-id: 90126cc714381e07, stream-reader-type: stream Message, ts: 2022-09-29T02:59:04.656Z}
04:59:04.678 CEST{caller: etcdserver/zap_raft.go:77, level: info, logger: raft, msg: 3ff1b5cd453a87df switched to configuration voters=(3177002992052145560 10381479693335928327), ts: 2022-09-29T02:59:04.653Z}
04:59:04.678 CEST{caller: membership/cluster.go:472, cluster-id: b0d7015fda1525c8, level: info, local-member-id: 3ff1b5cd453a87df, msg: removed member, removed-remote-peer-id: 3ff1b5cd453a87df, removed-remote-peer-urls: […], ts: 2022-09-29T02:59:04.657Z}
04:59:04.678 CEST{caller: rafthttp/peer_status.go:66, error: failed to dial 90126cc714381e07 on stream MsgApp v2 (the member has been permanently removed from the cluster), level: warn, msg: peer became inactive (message send to peer failed), peer-id: 90126cc714381e07, ts: 2022-09-29T02:59:04.659Z}
04:59:04.678 CEST{caller: etcdserver/server.go:1150, error: the member has been permanently removed from the cluster, level: warn, msg: server error, ts: 2022-09-29T02:59:04.659Z}
04:59:04.678 CEST{caller: etcdserver/server.go:1151, level: warn, msg: data-dir used by this member must be removed, ts: 2022-09-29T02:59:04.659Z}
04:59:04.678 CEST{caller: rafthttp/peer.go:330, level: info, msg: stopping remote peer, rem

etcd-2 节点日志 #

04:59:04.655 CEST{caller: rafthttp/stream.go:421, error: EOF, level: warn, local-member-id: 3ff1b5cd453a87df, msg: lost TCP streaming connection with remote peer, remote-peer-id: 90126cc714381e07, stream-reader-type: stream MsgApp v2, ts: 2022-09-29T02:59:04.654Z}
04:59:04.678 CEST{caller: rafthttp/stream.go:421, error: EOF, level: warn, local-member-id: 3ff1b5cd453a87df, msg: lost TCP streaming connection with remote peer, remote-peer-id: 90126cc714381e07, stream-reader-type: stream Message, ts: 2022-09-29T02:59:04.656Z}
04:59:04.678 CEST{caller: etcdserver/zap_raft.go:77, level: info, logger: raft, msg: 3ff1b5cd453a87df switched to configuration voters=(3177002992052145560 10381479693335928327), ts: 2022-09-29T02:59:04.653Z}
04:59:04.678 CEST{caller: membership/cluster.go:472, cluster-id: b0d7015fda1525c8, level: info, local-member-id: 3ff1b5cd453a87df, msg: removed member, removed-remote-peer-id: 3ff1b5cd453a87df, removed-remote-peer-urls: […], ts: 2022-09-29T02:59:04.657Z}
04:59:04.678 CEST{caller: rafthttp/peer_status.go:66, error: failed to dial 90126cc714381e07 on stream MsgApp v2 (the member has been permanently removed from the cluster), level: warn, msg: peer became inactive (message send to peer failed), peer-id: 90126cc714381e07, ts: 2022-09-29T02:59:04.659Z}
04:59:04.678 CEST{caller: etcdserver/server.go:1150, error: the member has been permanently removed from the cluster, level: warn, msg: server error, ts: 2022-09-29T02:59:04.659Z}
04:59:04.678 CEST{caller: etcdserver/server.go:1151, level: warn, msg: data-dir used by this member must be removed, ts: 2022-09-29T02:59:04.659Z}
04:59:04.678 CEST{caller: rafthttp/peer.go:330, level: info, msg: stopping remote peer, remote-peer-id: 2c16fb63879f0d98, ts: 2022-09-29T02:59:04.660Z}

请求帮助 #

已多次观察到该故障，无法定位具体原因，怀疑是GKE集群层面的问题，但不知如何排查解决，恳请协助分析处理。

内容的提问来源于stack exchange，提问作者Wolfgang Friedl