如何解决SpringCloud Gateway的Connection reset by peer错误？

在版本2.6.6的SpringBoot应用上执行Jmeter并发测试（并发数=5），使用SpringCloud Gateway 3.1.2版本时，网关频繁抛出Connection reset by peer错误，相关日志如下：

2022-11-21 15:39:01.541  WARN 218 --- [ctor-http-nio-2] r.n.h.c.HttpClientConnect                : [86ec2bb9-8] The connection observed an error, the request cannot be retried as the headers/body were sent
java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.8.0_322]
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.8.0_322]
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.8.0_322]
    at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[?:1.8.0_322]
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) ~[?:1.8.0_322]
    at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:258) ~[netty-buffer-4.1.77.Final.jar!/:4.1.77.Final]
    at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[netty-buffer-4.1.77.Final.jar!/:4.1.77.Final]
    at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357) ~[netty-transport-4.1.77.Final.jar!/:4.1.77.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151) ~[netty-transport-4.1.77.Final.jar!/:4.1.77.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[netty-transport-4.1.77.Final.jar!/:4.1.77.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658) ~[netty-transport-4.1.77.Final.jar!/:4.1.77.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584) ~[netty-transport-4.1.77.Final.jar!/:4.1.77.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[netty-transport-4.1.77.Final.jar!/:4.1.77.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[netty-common-4.1.77.Final.jar!/:4.1.77.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.77.Final.jar!/:4.1.77.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.77.Final.jar!/:4.1.77.Final]
    at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_322]
2022-11-21 15:39:01.542  WARN 218 --- [ctor-http-nio-2] c.h.h.f.g.c.AbstractJalorDiscoveryClient : put unhealthy instance to cache, instanceId: xxx:ccc33637fb9d406ab9ba4c92401c04a1-848d7c89b6-vthhr:52766
2022-11-21 15:39:01.542 ERROR 218 --- [ctor-http-nio-2] c.h.h.f.g.f.ServerFilter                 : [ServerResponse]{requestId=0462eed0-383780, traceId=a054ca3222698b572bf64c1f857df8cf, ServerInfo[instanceId=*.*.7.199:52766, startTime=1669016341539, endTime=1669016341542, duration=3, statusCode=500, exception=Connection reset by peer]}
2022-11-21 15:39:01.542 ERROR 218 --- [ctor-http-nio-2] c.h.h.f.g.f.ServerExceptionFilter        : [ServerResponse]{requestId=0462eed0-383780, traceId=a054ca3222698b572bf64c1f857df8cf, error=Connection reset by peer}
2022-11-21 15:39:01.542 ERROR 218 --- [ctor-http-nio-2] .h.f.g.e.GatewayErrorWebExceptionHandler : requestId=0462eed0-383780, traceId=a054ca3222698b572bf64c1f857df8cf, error={timestamp=2022-11-21T15:39:01.542+0800, path=xxx, status=500, error=Internal Server Error, requestId=0462eed0-383780}
2022-11-21 15:39:01.542 ERROR 218 --- [ctor-http-nio-2] a.w.r.e.AbstractErrorWebExceptionHandler : [0462eed0-383780]  500 Server Error for HTTP POST "xxx"

已尝试以下配置但未解决问题：

• 网关参数：spring.cloud.gateway.httpclient.pool.max-idle-time=10000
• SpringBoot系统参数：server.tomcat.connection-timeout=20000

另外发现，当测试为串行模式（并发数=1）时，几乎不会出现该错误。

1. 完善网关连接池核心配置 #

仅配置空闲超时不足以应对并发场景，需补充连接池的最大连接数、活跃连接超时、连接获取超时等参数：

# 网关HTTP客户端连接池最大连接数
spring.cloud.gateway.httpclient.pool.max-connections=200
# 活跃连接最长存活时间（毫秒），到期自动回收
spring.cloud.gateway.httpclient.pool.max-life-time=60000
# 从连接池获取连接的超时时间（毫秒）
spring.cloud.gateway.httpclient.connect-timeout=5000

2. 开启连接池健康检查 #

开启Netty HttpClient的连接健康检查，自动剔除后端已断开的失效连接，避免复用无效连接：

# 启用连接池健康检查
spring.cloud.gateway.httpclient.pool.health-check.enabled=true
# 健康检查执行间隔（毫秒）
spring.cloud.gateway.httpclient.pool.health-check.interval=30000

3. 排查后端服务的连接配置 #

Connection reset by peer多由后端主动断开连接导致，需检查后端服务：

• Tomcat连接配置：调整maxKeepAliveRequests（长连接最大请求数）、keepAliveTimeout（长连接空闲超时），避免后端提前关闭长连接
• 并发处理能力：检查后端线程池参数（如Tomcat的maxThreads、acceptCount），确保能承载并发请求
• 中间设备配置：查看防火墙、负载均衡器是否存在主动断开空闲连接的规则，调整超时时间与网关匹配

4. 配置针对性的请求重试策略 #

针对500错误配置重试逻辑，注意仅对幂等请求（如GET）或确保幂等的POST请求开启重试：

spring:
  cloud:
    gateway:
      routes:
        - id: your_route_id
          uri: lb://your-service
          predicates:
            - Path=/xxx/**
          filters:
            - name: Retry
              args:
                retries: 2
                statuses: INTERNAL_SERVER_ERROR
                methods: GET,POST
                backoff:
                  firstBackoff: 100ms
                  maxBackoff: 500ms
                  factor: 2
                  basedOnPreviousValue: false

5. 升级依赖版本 #

当前使用的Spring Boot 2.6.6和Spring Cloud Gateway 3.1.2存在部分已知的连接池管理bug，建议升级到兼容的稳定补丁版本：

• Spring Boot 2.6.x最新补丁版（如2.6.14）
• Spring Cloud Gateway 3.1.x最新补丁版（如3.1.6）

内容的提问来源于stack exchange，提问作者zhu mo