Solr Cloud日志出现大量报错但无不良影响，是否需要排查处理？

报错核心定位 #

报错根因为java.net.ConnectException: Connection refused，是questions_shard7_replica_n80节点转发写入请求到同集群的questions_shard6_replica_n70副本时，无法连接目标节点的8983端口触发的异常。
完整报错日志如下：

8/24/2021, 4:42:25 PM
ERROR true
x:questions_shard7_replica_n80
ErrorReportingConcurrentUpdateSolrClient
调用SolrCmdDistributor$Req时出错：cmd=add{,​id=q36445451}; node=ForwardNode: http://localhost:8983/solr/questions_shard6_replica_n70/ to http://localhost:8983/solr/questions_shard6_replica_n70/
java.io.IOException: java.net.ConnectException: Connection refused
at org.eclipse.jetty.client.util.DeferredContentProvider.flush(DeferredContentProvider.java:197)
at org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.flush(OutputStreamContentProvider.java:151)
at org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.write(OutputStreamContentProvider.java:145)
at org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:216)
at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:209)
at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:172)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:106)
at org.apache.solr.client.solrj.impl.BinaryRequestWriter.write(BinaryRequestWriter.java:83)
at org.apache.solr.client.solrj.impl.Http2SolrClient.send(Http2SolrClient.java:342)
at org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:237)
at org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:181)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Suppressed: java.io.IOException: java.net.ConnectException: Connection refused
at org.eclipse.jetty.client.util.DeferredContentProvider.flush(DeferredContentProvider.java:197)
at org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.flush(OutputStreamContentProvider.java:151)
at org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.write(OutputStreamContentProvider.java:145)
at org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:216)
at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:209)
at org.apache.solr.common.util.JavaBinCodec.close(JavaBinCodec.java:1277)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:107)
... 9 more
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:774)
at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:120)
at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:397)
at org.eclipse.jetty.http2.client.http.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:146)
at org.eclipse.jetty.http2.client.http.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:141)
at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:620)
at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:597)
at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:590)
at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:186)
... 4 more
Caused by: [CIRCULAR REFERENCE: java.net.ConnectException: Connection refused]

暂时无业务影响的原因 #

Solr Cloud默认有多副本冗余机制：

• 写入请求转发到异常节点失败后，会自动重试同分片的其他健康副本，只要分片内至少有1个健康的Leader副本+1个正常副本，写入就不会失败
• 查询请求只会路由到状态正常的副本节点，不会访问异常节点，因此对外查询服务也无感知

存在的隐患 #

• 分片冗余度下降：当前questions_shard6分片少了一个可用副本，若同分片其他副本再出现故障，会直接导致分片不可用，业务出现写入/查询失败
• 集群资源浪费：大量失败请求会触发内部重试逻辑，额外消耗CPU、网络带宽资源，同时大量报错日志会持续占用磁盘存储空间，高并发场景下可能引发性能波动甚至雪崩
• 数据恢复风险：异常节点离线期间的更新数据会积压在其他副本的事务日志中，异常节点离线时间越长，后续恢复时的索引同步耗时越久，甚至可能出现同步失败、数据不一致的问题

排查处理建议 #

• 首先登录questions_shard6_replica_n70所在的服务器，执行ps -ef | grep solr检查Solr进程是否存活，执行nc -zv localhost 8983检测8983端口是否正常监听，排查是否是进程崩溃、防火墙拦截、端口被占用导致的连接失败
• 检查目标节点的Solr GC日志、系统日志，确认是否存在长时间Full GC导致进程假死、OOM被系统主动kill的情况
• 访问Solr Admin UI的Cloud->Nodes页面，确认questions_shard6_replica_n70的状态，如果处于Recovery失败状态，可先删除该异常副本，再触发集群自动在对应节点重建副本即可
• 处理完成后观察30分钟集群日志，确认无同类报错后再恢复正常运维巡检

内容的提问来源于stack exchange，提问作者Alex Benjamen