Istio与Spring Cloud Gateway共存时WebSocket握手502错误如何解决
我们的技术栈包含OpenShift、Istio、Cloud Gateway微服务及多个业务微服务,使用的框架版本如下:
- Spring-boot 2.3.6.RELEASE
- Spring-Cloud Hoxton.SR9
- Spring-Cloud-Starter-Gateway 2.2.6.RELEASE
当前遇到的问题为WebSocket握手失败,错误信息为Invalid handshake response getStatus: 502 Bad Gateway。经排查验证:剔除Istio时WebSocket功能正常;使用Istio但剔除Cloud Gateway时功能也可正常运行。初步判断问题原因是Istio会添加Content-length=0请求头,导致Cloud Gateway处理WebSocket握手流程失败。
application.yml 配置
spring: profiles: active: default cloud: discovery: reactive: enabled: false kubernetes: discovery: enabled: false gateway: streaming-media-types: - application/stream+json - text/event-stream - application/javascript routes: - id: microservice_name_websocket_sockjs_route uri: ${MICROSERVICE_NAME_SOCKJS_URI:lb://microservice-name} predicates: - Path=/microservice-name/ui/groupoperations/tasks/** filters: - RewritePath=/microservice-name/(?<segment>.*), /$\{segment} - id: microservice_name_webSocketRoute uri: ${MICROSERVICE_NAME_WEBSOCKET_URI:lb:ws://microservice-name} predicates: - Path=/microservice-name/ui/groupoperations/tasks/** filters: - RewritePath=/microservice-name/(?<segment>.*), /$\{segment} server: port: 8889 ribbon: MaxAutoRetries: 1 MaxAutoRetriesNextServer: 1 ReadTimeout: 2000 ConnectTimeout: 1000 NIWSServerListClassName: io.fabric8.kubeflix.ribbon.KubernetesServerList PortName: http logging: level: ROOT: TRACE com.corporation.project: TRACE org.springframework: TRACE reactor.netty.http.client: TRACE white-list-filter: file: location: ${WHITE_LIST_FILE_LOCATION:classpath:filters} fileName: ${WHITE_LIST_FILE_NAME:cloud-gateway-whitelist.list} management: security: enabled: false health: jms: enabled: false endpoint: metrics: enabled: true prometheus: enabled: true endpoints: web: exposure: include: '*' base-path: / metrics: export: prometheus: enabled: true exception-control: enable: false forbidden-words-regex: (stack)|(exception)|(select)|(update)|(delete)|(insert) action: log_only status-to-replace: 418 body-to-replace: 'Data leak prevented' opentracing: jaeger: service-name: cloud-gateway enable-b3-propagation: true log-spans: true spring: cloud: jdbc: enabled: false
异常栈信息
2021-09-05 16:28:14.779 TRACE 1 --- [or-http-epoll-1] o.s.w.s.adapter.HttpWebHandlerAdapter : [2a7a738e-7] Completed 200 OK, headers={masked} 2021-09-05 16:28:14.779 TRACE 1 --- [or-http-epoll-1] o.s.h.s.r.ReactorHttpHandlerAdapter : [2a7a738e-7] Handling completed 2021-09-05 16:28:14.781 DEBUG 1 --- [or-http-epoll-4] r.netty.http.client.HttpClientConnect : [id: 0x0c5a1b20, L:/1.2.15.231:56854 - R:microservice-name/1.8.15.171:8080] Handler is being applied: {uri=ws://microservice-name:8080/ui/gp/t/516/tzc151tm/websocket, method=GET} 2021-09-05 16:28:14.803 DEBUG 1 --- [or-http-epoll-4] r.n.http.client.HttpClientOperations : [id: 0x0c5a1b20, L:/1.2.15.231:56854 - R:microservice-name/1.8.15.171:8080] Attempting to perform websocket handshake with ws://cloud-gateway.ob3dl-opa1-devel.svc.cluster.local:8889/ui/gp/t/516/tzc151tm/websocket 2021-09-05 16:28:14.828 DEBUG 1 --- [or-http-epoll-4] i.n.h.c.h.w.WebSocketClientHandshaker13 : WebSocket version 13 client handshake key: I+YlMp+qLtlj4iBhDlsBzg==, expected response: FK9mNuXf/KDCCE3Qyelp7dBdbZ8= 2021-09-05 16:28:14.891 WARN 1 --- [or-http-epoll-4] r.netty.http.client.HttpClientConnect : [id: 0x0c5a1b20, L:/1.2.15.231:56854 - R:microservice-name/1.8.15.171:8080] The connection observed an error io.netty.handler.codec.http.websocketx.WebSocketClientHandshakeException: Invalid handshake response getStatus: 502 Bad Gateway at io.netty.handler.codec.http.websocketx.WebSocketClientHandshaker13.verify(WebSocketClientHandshaker13.java:272) ~[netty-codec-http-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.handler.codec.http.websocketx.WebSocketClientHandshaker.finishHandshake(WebSocketClientHandshaker.java:302) ~[netty-codec-http-4.1.54.Final.jar!/:4.1.54.Final] at reactor.netty.http.client.WebsocketClientOperations.onInboundNext(WebsocketClientOperations.java:116) ~[reactor-netty-0.9.21.RELEASE.jar!/:0.9.21.RELEASE] at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:96) ~[reactor-netty-0.9.21.RELEASE.jar!/:0.9.21.RELEASE] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) ~[netty-codec-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) ~[netty-codec-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) ~[netty-transport-native-epoll-4.1.54.Final-linux-x86_64.jar!/:4.1.54.Final] at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) ~[netty-transport-native-epoll-4.1.54.Final-linux-x86_64.jar!/:4.1.54.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) ~[netty-transport-native-epoll-4.1.54.Final-linux-x86_64.jar!/:4.1.54.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.54.Final.jar!/:4.1.54.Final] at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
截图说明
WebSocket响应截图,Messages标签页为空
根因说明
该问题是Spring Cloud Gateway Hoxton版本的已知兼容性问题:Istio sidecar在代理请求时会为无请求体的GET请求默认添加Content-Length: 0请求头,而该版本的Spring Cloud Gateway处理WebSocket握手请求时,未对该头做特殊处理,会按普通带body的请求逻辑处理,最终导致握手失败返回502。
方案1:修改Gateway路由配置(推荐,无需改动代码)
在现有的两个WebSocket相关路由的filters配置中,添加移除Content-Length请求头的规则即可,修改后的路由配置示例:
routes: - id: microservice_name_websocket_sockjs_route uri: ${MICROSERVICE_NAME_SOCKJS_URI:lb://microservice-name} predicates: - Path=/microservice-name/ui/groupoperations/tasks/** filters: - RewritePath=/microservice-name/(?<segment>.*), /$\{segment} - RemoveRequestHeader=Content-Length - id: microservice_name_webSocketRoute uri: ${MICROSERVICE_NAME_WEBSOCKET_URI:lb:ws://microservice-name} predicates: - Path=/microservice-name/ui/groupoperations/tasks/** filters: - RewritePath=/microservice-name/(?<segment>.*), /$\{segment} - RemoveRequestHeader=Content-Length
方案2:自定义全局过滤器(适合批量WebSocket路由场景)
如果服务内存在大量WebSocket路由,不想逐个修改配置,可以新增一个全局过滤器,自动识别WebSocket握手请求并移除Content-Length头,代码示例如下:
import org.springframework.cloud.gateway.filter.GatewayFilterChain; import org.springframework.cloud.gateway.filter.GlobalFilter; import org.springframework.http.HttpHeaders; import org.springframework.http.server.reactive.ServerHttpRequest; import org.springframework.stereotype.Component; import org.springframework.web.server.ServerWebExchange; import reactor.core.publisher.Mono; @Component public class WebsocketContentLengthCleanFilter implements GlobalFilter { @Override public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) { ServerHttpRequest request = exchange.getRequest(); String upgradeHeader = request.getHeaders().getFirst(HttpHeaders.UPGRADE); // 识别WebSocket握手请求 if ("websocket".equalsIgnoreCase(upgradeHeader)) { ServerHttpRequest modifiedRequest = request.mutate() .headers(httpHeaders -> httpHeaders.remove(HttpHeaders.CONTENT_LENGTH)) .build(); return chain.filter(exchange.mutate().request(modifiedRequest).build()); } return chain.filter(exchange); } }
方案3:升级Spring Cloud版本(长期兼容方案)
该问题已在Spring Cloud 2020.0.x(ILford)及之后的版本中官方修复,新版本会自动处理WebSocket握手请求的Content-Length头,无需额外配置。如果业务允许升级框架版本,可以直接升级到兼容的版本即可。
内容的提问来源于stack exchange,提问作者Alexandr Shuvaev

