Spring Boot Actuator中Elasticsearch健康检查偶发Down状态求助

看你遇到的这个问题——Spring Boot 2.0.2应用用Transport Client连ES 6.2.4，健康检查每天随机触发6-8次超时告警，无固定规律。先结合你贴的报错日志分析下：核心是org.elasticsearch.ElasticsearchTimeoutException，触发点在Actuator的ElasticsearchHealthIndicator执行健康检查时，同步调用actionGet()超时了。给你几个实用的解决思路：

1. 自定义健康检查超时时间 #

默认的ElasticsearchHealthIndicator超时设置偏短，当ES集群临时负载高时很容易卡壳。你可以自己写个健康指示器，手动设置更长的超时时间：

import org.elasticsearch.action.admin.cluster.health.ClusterHealthRequest;
import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse;
import org.elasticsearch.client.transport.TransportClient;
import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.stereotype.Component;

@Component
public class CustomElasticsearchHealthIndicator implements HealthIndicator {

    private final TransportClient transportClient;

    public CustomElasticsearchHealthIndicator(TransportClient transportClient) {
        this.transportClient = transportClient;
    }

    @Override
    public Health health() {
        try {
            // 这里设置5秒超时，你可以根据实际情况调整
            ClusterHealthResponse response = transportClient.admin().cluster()
                    .health(new ClusterHealthRequest()).actionGet(5000);
            return Health.up()
                    .withDetail("clusterName", response.getClusterName())
                    .withDetail("status", response.getStatus().name())
                    .withDetail("numberOfNodes", response.getNumberOfNodes())
                    .build();
        } catch (Exception e) {
            return Health.down(e).build();
        }
    }
}

替换默认的健康检查逻辑后，给ES足够的响应缓冲时间，能有效减少随机超时的情况。

2. 优化Transport Client的连接配置 #

检查下你的Transport Client配置，调整以下参数提升稳定性：

• client.transport.ping_timeout：延长节点ping超时，比如设为30秒
• client.transport.nodes_sampler_interval：拉长节点采样间隔，比如设为5分钟
• 确保连接池有足够容量，同时开启节点嗅探功能

示例配置类：

import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.transport.TransportAddress;
import org.elasticsearch.transport.client.PreBuiltTransportClient;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.net.InetAddress;
import java.net.UnknownHostException;

@Configuration
public class ElasticsearchConfig {

    @Bean
    public TransportClient transportClient() throws UnknownHostException {
        Settings settings = Settings.builder()
                .put("cluster.name", "your-cluster-name")
                .put("client.transport.ping_timeout", "30s")
                .put("client.transport.nodes_sampler_interval", "5m")
                .put("client.transport.sniff", true)
                .build();

        TransportClient client = new PreBuiltTransportClient(settings)
                .addTransportAddress(new TransportAddress(InetAddress.getByName("es-host"), 9300));
        return client;
    }
}

3. 排查Elasticsearch集群状态 #

随机超时大概率和ES集群的临时负载有关，建议排查这些点：

• 查看ES的GC日志，确认是否有频繁Full GC导致服务停顿
• 开启ES慢查询日志，检查是否有耗时过长的查询占用资源
• 监控ES集群的CPU、内存、磁盘IO指标，看超时时段是否有资源瓶颈
• 确认超时时段是否在执行分片迁移、索引刷新等后台操作

4. 迁移到Rest High Level Client #

Transport Client在ES 6.x已经被标记为弃用，7.x版本彻底移除了这个客户端。Rest High Level Client是官方推荐的替代方案，兼容性和稳定性更好，而且Spring Boot 2.x也提供了完善的自动配置支持。长远来看，迁移到Rest Client能从根本上减少这类兼容性问题。

附上你提供的报错日志：

2020-03-18 21:18:12.048 WARN 3924 --- [nio-8082-exec-5] o.s.b.a.e.ElasticsearchHealthIndicator : Elasticsearch health check failed
org.elasticsearch.ElasticsearchTimeoutException: java.util.concurrent.TimeoutException: Timeout waiting for task.
 at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:71)
 at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:58)
 at org.springframework.boot.actuate.elasticsearch.ElasticsearchHealthIndicator.doHealthCheck(ElasticsearchHealthIndicator.java:80)
 at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:84)
 at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:68)
 at org.springframework.boot.actuate.health.HealthEndpointWebExtension.getHealth(HealthEndpointWebExtension.java:50)
 at sun.reflect.GeneratedMethodAccessor76.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:223)
 at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:76)
 at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:61)
 at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$ServletWebOperationAdapter.handle(AbstractWebMvcEndpointHandlerMapping.java:243)
 at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(AbstractWebMvcEndpointHandlerMapping.java:299)
 at sun.reflect.GeneratedMethodAccessor75.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209)
 at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
 at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
 at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:877)
 at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:783)
 at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
 at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991)
 at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925)
 at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974)
 at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:866)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
 at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
 at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
 at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:158)
 at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:126)
 at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:111)
 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
 at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:84)
 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
 at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
 at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
 at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
 at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
 at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
 at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
 at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
 at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
 at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1468)
 at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
 at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException: Timeout waiting for task.
 at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:232)
 at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:67)
 at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:69)
 ... 76 common frames omitted

内容的提问来源于stack exchange，提问作者Vinit Pillai