Spring Cloud Gateway断路器出现java.util.concurrent.TimeoutException异常排查

版本信息（必须使用Java8） #

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.7.3</version>
</parent>

<properties>
    <java.version>1.8</java.version>
    <spring-cloud.version>2021.0.5</spring-cloud.version>
</properties>

复现问题的application.yml配置 #

spring:
  cloud:
    gateway:
      routes:
        - id: test
          order: 0
          uri: http://{ANY_SITE_CAUSING_CONNECTION_TIMEOUT}:8080/
          predicates:
            - Path=/**
          filters:
            - name: CircuitBreaker
              args:
                name: cbTest
                fallbackUri: forward:/fallback

management:
# 用于通过actuator查看断路器状态，已省略

resilience4j:
  circuitbreaker:
    configs:
      default:
        registerHealthIndicator: true
        minimumNumberOfCalls: 10
    instances:
      cbTest:
        baseConfig: default

Fallback控制器代码（用于查看异常） #

@RestController
@RequestMapping("/fallback")
public class FallbackController {
    @GetMapping
    public Mono<Void> fallback(ServerWebExchange exchange) {
        Throwable t = exchange.getAttribute(ServerWebExchangeUtils.CIRCUITBREAKER_EXECUTION_EXCEPTION_ATTR);
        Objects.requireNonNull(t).printStackTrace();
        return null;
    }
}

异常表现 #

添加断路器后，请求在1秒内结束，抛出意外的java.util.concurrent.TimeoutException，而非预期的连接超时：

java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'circuitBreaker' (and no fallback has been configured)
    at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:295)
    at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:280)
    at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:419)
    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:271)
    at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:286)
    at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
    at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
    at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
    at java.util.concurrent.FutureTask.run(FutureTask.java)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

断路器事件记录 #

通过http://localhost:8080/actuator/circuitbreakerevents查看的事件：

{
    "circuitBreakerEvents": [
        {
            "circuitBreakerName": "cbTest",
            "type": "ERROR",
            "creationTime": "2023-02-14T10:21:12.633+09:00[Asia/Seoul]",
            "errorMessage": "java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'circuitBreaker' (and no fallback has been configured)",
            "durationInMs": 1000,
            "stateTransition": null
        }
    ]
}

移除断路器后的预期表现 #

移除路由配置中的filters部分后，请求会尝试连接30秒，抛出预期的Netty连接超时异常：

io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection timed out: no further information: /{ANY_SITE_CAUSING_CONNECTION_TIMEOUT}:8080
    Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
    *__checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ HTTP GET "/x" [ExceptionHandlingWebHandler]
Original Stack Trace:
Caused by: java.net.ConnectException: Connection timed out: no further information
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_292]
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) ~[na:1.8.0_292]
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) ~[netty-transport-4.1.79.Final.jar:4.1.79.Final]
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[netty-transport-4.1.79.Final.jar:4.1.79.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:710) ~[netty-transport-4.1.79.Final.jar:4.1.79.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658) ~[netty-transport-4.1.79.Final.jar:4.1.79.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584) ~[netty-transport-4.1.79.Final.jar:4.1.79.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[netty-transport-4.1.79.Final.jar:4.1.79.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.79.Final.jar:4.1.79.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.79.Final.jar:4.1.79.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.79.Final.jar:4.1.79.Final]
    at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_292]

疑问 #

为何Netty的连接超时异常会被1000ms的java.util.concurrent.TimeoutException覆盖？尝试增大spring.cloud.gateway.httpclient.connect-timeout、spring.cloud.gateway.httpclient.response-timeout的值，以及配置路由级别的超时，均无效果。

内容的提问来源于stack exchange，提问作者Shoner Sul