触发RefreshEvent后Eureka与Ribbon负载均衡异常排查求助

环境与实现代码 #

应用依赖父版本：

<parent>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-netflix</artifactId>
    <version>2.2.5.RELEASE</version>
</parent>

已实现自定义Ribbon负载均衡规则，核心代码如下：

Ribbon配置类 #

@RibbonClients(defaultConfiguration = CustomizedRibbonConfig.class)
public class RibbonClientConfiguration {
    public static class BazServiceList extends ConfigurationBasedServerList {
        public BazServiceList(IClientConfig config) {
            super.initWithNiwsConfig(config);
        }
    }
}

@Configuration
class CustomizedRibbonConfig {
    @Bean
    public IRule ribbonRule() {
        return new MetadataAwareRule();
    }

    @Bean
    public ServerListUpdater ribbonServerListUpdater() {
        return new EurekaNotificationServerListUpdater();
    }
}

自定义Predicate #

public class MetadataAwarePredicate extends AbstractDiscoveryEnabledPredicate {

    /**
     * {@inheritDoc}
     */
    @Override
    protected boolean apply(DiscoveryEnabledServer server) {
        return true;
    }
}

自定义负载均衡规则 #

@Slf4j
public class MetadataAwareRule extends AbstractDiscoveryEnabledRule {
    public static final ThreadLocal<String> CURRENT_LOAD_BALANCED_SERVICE_IP = new ThreadLocal<>();

    /**
     * Creates new instance of {@link MetadataAwareRule}.
     */
    public MetadataAwareRule() {
        this(new MetadataAwarePredicate());
    }

    /**
     * Creates new instance of {@link MetadataAwareRule} with specific predicate.
     *
     * @param predicate the predicate, can't be {@code null}
     * @throws IllegalArgumentException if predicate is {@code null}
     */
    public MetadataAwareRule(AbstractDiscoveryEnabledPredicate predicate) {
        super(predicate);
    }


    @Override
    public Server choose(Object key) {
       ....my customized choose policy....
}

触发方式 #

通过以下接口发布RefreshEvent触发应用刷新：

@GetMapping("/test/event")
public CommonResult testRaiseRefreshEvent() {
    ApplicationContextHolder.getApplicationContext().publishEvent(new RefreshEvent(this, null, "test to trigger the problem"));
    return CommonResult.succeed();
}

触发后出现的两类互斥异常 #

异常类型1：Eureka缓存刷新失败，负载均衡无可用服务 #

触发后抛出RedirectingEurekaHttpClient请求执行错误，伴随JSON解析失败，无法刷新Eureka缓存：

2022-10-19 11:29:32.947 [app:web-gateway,traceId:,spanId:,parentId:] [DiscoveryClient-CacheRefreshExecutor-0] ERROR | RedirectingEurekaHttpClient.java:83 | c.n.d.s.t.d.RedirectingEurekaHttpClient | Request execution error. endpoint=DefaultEndpoint{ serviceUrl='http://localhost:8000/eureka/}
javax.ws.rs.WebApplicationException: com.fasterxml.jackson.core.JsonParseException: processing aborted
 at [Source: (GZIPInputStream); line: 1, column: 18]
    at com.netflix.discovery.provider.DiscoveryJerseyProvider.readFrom(DiscoveryJerseyProvider.java:110)
    ... 69 common frames omitted

同时出现DiscoveryClient缓存刷新失败：

2022-10-19 11:29:32.956 [app:web-gateway,traceId:,spanId:,parentId:] [DiscoveryClient-CacheRefreshExecutor-0] ERROR | DiscoveryClient.java:1018 | c.netflix.discovery.DiscoveryClient | DiscoveryClient_WEB-GATEWAY/192.168.56.1:web-gateway:8004:NEW_GATEWAY_DEFAULT_GROUP - was unable to refresh its cache! status = Cannot execute request on any known server
com.netflix.discovery.shared.transport.TransportException: Cannot execute request on any known server
    at com.netflix.discovery.shared.transport.decorator.RetryableEurekaHttpClient.execute(RetryableEurekaHttpClient.java:112)
    ... 69 common frames omitted

后续请求因负载均衡无可用服务失败：

java.lang.RuntimeException: com.netflix.client.ClientException: Load balancer does not have available server for client: web-message-center
    at org.springframework.cloud.openfeign.ribbon.LoadBalancerFeignClient.execute(LoadBalancerFeignClient.java:90)
    ... 15 common frames omitted

异常类型2：EurekaNotificationServerListUpdater更新线程终止，服务列表无法更新 #

刷新后EurekaNotificationServerListUpdater的服务器列表更新线程被终止，无法获取Eureka最新服务信息；即使目标服务下线，网关仍会转发请求，而非返回服务不可用提示（已配置该异常的处理器）。

注意：以上两类问题仅在触发RefreshEvent后出现，且不会同时发生，请求帮忙排查问题原因或提供排查方向。

内容的提问来源于stack exchange，提问作者william