ELK集群elk01节点故障求助：版本不兼容无法加入集群

问题现象 #

拥有elk01、elk02、elk03三节点的ELK集群中，elk01突然宕机，重启后无法加入集群。查看elk01节点日志/var/log/elasticsearch/elasticsearch.log，出现如下认证错误：

"[elk01] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]"
"[elk01] Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]"

已执行的排查操作 #

• 重启所有节点的Elasticsearch服务
• 所有节点的9200、9300端口可通过telnet正常连通
• 在elk01节点执行密码重置命令/usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u elastic，报错Error: Failed to determine the health of the cluster.
• 在所有节点的elasticsearch.yml中设置xpack.security.enabled: false并重启服务，仍无法完成密码重置
• 从elk02或elk03节点执行curl http://elk02:9200/_cat/indices?v，确认所有索引状态为green

核心错误日志（翻译后） #

更新elk01节点日志后发现根本原因：节点版本8.5.0与集群内索引.monitoring-es-7-2023.09.14的8.5.2版本不兼容，导致elk01无法加入集群，具体日志如下：

[2023-09-14T23:10:09,174][INFO ][o.e.c.c.JoinHelper       ] [elk01] 尝试加入集群节点{elk03}{lNN-V6I5S3mgyAOL1_taXg}{EM6lZBPcTpK4lqwdIq8udA}{elk03}{xx.xx.xx.224}{xx.xx.xx.224:9300}{cdfhilmrstw}{ml.allocated_processors_double=4.0, ml.machine_memory=16526131200, xpack.installed=true, ml.max_jvm_size=8262778880, ml.allocated_processors=4}失败，JoinRequest详情：{sourceNode={elk01}{twbA5ovpSP-gAkeD65cmNg}{62VkkaWSR86H7-dlzTn1xg}{elk01}{xx.xx.xx.222}{xx.xx.xx.222:9300}{cdfhilmrstw}{ml.max_jvm_size=1073741824, ml.allocated_processors_double=8.0, xpack.installed=true, ml.machine_memory=16525099008, ml.allocated_processors=8}, minimumTerm=2609, optionalJoin=Optional[Join{term=2609, lastAcceptedTerm=10, lastAcceptedVersion=4237, sourceNode={elk01}{twbA5ovpSP-gAkeD65cmNg}{62VkkaWSR86H7-dlzTn1xg}{elk01}{xx.xx.xx.222}{xx.xx.xx.222:9300}{cdfhilmrstw}{ml.max_jvm_size=1073741824, ml.allocated_processors_double=8.0, xpack.installed=true, ml.machine_memory=16525099008, ml.allocated_processors=8}, targetNode={elk03}{lNN-V6I5S3mgyAOL1_taXg}{EM6lZBPcTpK4lqwdIq8udA}{elk03}{xx.xx.xx.224}{xx.xx.xx.224:9300}{cdfhilmrstw}{ml.allocated_processors_double=4.0, ml.machine_memory=16526131200, xpack.installed=true, ml.max_jvm_size=8262778880, ml.allocated_processors=4}}]}
org.elasticsearch.transport.RemoteTransportException: [elk03][xx.xx.xx.224:9300][internal:cluster/coordination/join]
Caused by: java.lang.IllegalStateException: 索引[.monitoring-es-7-2023.09.14/arNavb0JT92WnAajJMCyWQ]版本不被支持：8.5.2，当前节点版本为：8.5.0
	at org.elasticsearch.cluster.coordination.JoinTaskExecutor.ensureIndexCompatibility(JoinTaskExecutor.java:268) ~[elasticsearch-8.5.0.jar:?]
	at org.elasticsearch.cluster.coordination.JoinTaskExecutor.lambda$addBuiltInJoinValidators$9(JoinTaskExecutor.java:341) ~[elasticsearch-8.5.0.jar:?]
	at org.elasticsearch.cluster.coordination.Coordinator.lambda$validateJoinRequest$13(Coordinator.java:663) ~[elasticsearch-8.5.0.jar:?]
	at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
	at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1092) ~[?:?]
	at org.elasticsearch.cluster.coordination.Coordinator.validateJoinRequest(Coordinator.java:663) ~[elasticsearch-8.5.0.jar:?]
	at org.elasticsearch.cluster.coordination.Coordinator$1.onResponse(Coordinator.java:609) ~[elasticsearch-8.5.0.jar:?]
	at org.elasticsearch.cluster.coordination.Coordinator$1.onResponse(Coordinator.java:604) ~[elasticsearch-8.5.0.jar:?]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) ~[elasticsearch-8.5.0.jar:?]
	at org.elasticsearch.transport.ClusterConnectionManager.lambda$connectToNodeOrRetry$1(ClusterConnectionManager.java:146) ~[elasticsearch-8.5.0.jar:?]
	at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:245) ~[elasticsearch-8.5.0.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113) ~[elasticsearch-8.5.0.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:100) ~[elasticsearch-8.5.0.jar:?]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:131) ~[elasticsearch-8.5.0.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:139) ~[elasticsearch-8.5.0.jar:?]
	at org.elasticsearch.transport.ClusterConnectionManager.lambda$connectToNodeOrRetry$4(ClusterConnectionManager.java:253) ~[elasticsearch-8.5.0.jar:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) ~[elasticsearch-8.5.0.jar:?]
	at org.elasticsearch.action.ActionListener$RunAfterActionListener.onResponse(ActionListener.java:367) ~[elasticsearch-8.5.0.jar:?]
	at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127) ~[elasticsearch-8.5.0.jar:?]
	at org.elasticsearch.transport.TransportService.lambda$handshake$6(TransportService.java:560) ~[elasticsearch-8.5.0.jar:?]
	at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:245) ~[elasticsearch-8.5.0.jar:?]
	at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:43) ~[elasticsearch-8.5.0.jar:?]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1362) ~[elasticsearch-8.5.0.jar:?]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1362) ~[elasticsearch-8.5.0.jar:?]
	at org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:369) ~[elasticsearch-8.5.0.jar:?]
	at org.elasticsearch.transport.InboundHandler$2.doRun(InboundHandler.java:361) ~[elasticsearch-8.5.0.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:892) ~[elasticsearch-8.5.0.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.5.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]

方案1：升级elk01节点至8.5.2版本 #

• 下载Elasticsearch 8.5.2安装包，替换elk01节点的现有版本
• 停止elk01的Elasticsearch服务，备份数据目录（默认路径/var/lib/elasticsearch）
• 安装8.5.2版本，保持elasticsearch.yml配置与其他节点一致
• 启动elk01服务，查看日志确认节点成功加入集群
• 恢复xpack.security.enabled: true配置，重启所有节点，验证认证功能正常

方案2：删除不兼容的监控索引 #

注意：删除监控索引会丢失该时间段的监控数据，请谨慎操作

• 在elk02或elk03节点执行命令删除目标索引：

curl -X DELETE http://elk02:9200/.monitoring-es-7-2023.09.14

• 启动elk01节点，查看日志确认节点成功加入集群
• 恢复xpack.security.enabled: true配置，重启所有节点，在正常运行的节点执行密码重置命令：/usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u elastic
• 验证集群状态及认证功能正常

内容的提问来源于stack exchange，提问作者Linux_Admin