关于okhttp3 StreamResetException(CANCEL)异常的排查问询

环境版本信息 #

• OpenJDK 11.0.18
• Nginx 1.14.0
• OkHttp3 3.8.0

异常现象 #

本地无Nginx/传统负载均衡环境下无任何报错；生产环境不定期无规律触发异常，表现为连续N次请求成功后出现一次报错，成功请求次数序列为：15,13,13,11,14,17,30,30,11,2,37,27,6,2,5,21,10,29,13,51。

已知信息与疑问 #

• 已配置Nginx的proxy_redirect off;和proxy_buffering off;
• 了解到Nginx 1.15版本存在相关HTTP/2 Bug，但不确定是否影响1.14版本

异常栈信息 #

root@sandbox-gateway-1b.teller-sandbox [~] # cat /srv/www/gateway/logs/spring.log |grep '13:52:27.746' -A 30 
2023-02-08 13:52:27.746 [https-jsse-nio-32752-exec-16] INFO  JavaUtilLogger : [principal : null | request-id : 66420bde-abbe-4e9e-ba89-2a5b6e6390e9] HTTP CALL EXCEPTION: https://sandbox-api.example.com/api/v1/auth/token 
okhttp3.internal.http2.StreamResetException: stream was reset: CANCEL 
at okhttp3.internal.http2.Http2Stream$FramingSource.read(Http2Stream.java:384) 
at okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.java:286) 
at okio.Buffer.writeAll(Buffer.java:1143) 
at okio.RealBufferedSource.readString(RealBufferedSource.java:203) 
at okhttp3.ResponseBody.string(ResponseBody.java:182) 
at common.restclient.HttpClient.doCall(HttpClient.java:92) 
at common.restclient.HttpClient.call(HttpClient.java:53) 
at common.restclient.RestClientImpl.postWithoutHeaders(RestClientImpl.java:45) 
at example.client.exampleClientImpl.login(exampleClientImpl.java:37) 
at com.dux.gateway.core.adapters.example.example.loginToexample(example.java:74) 
at com.dux.gateway.core.adapters.example.example.processTransaction(example.java:47) 
at com.dux.gateway.core.adapters.example.example.processTransaction(example.java:22) 
at com.dux.gateway.core.TransactionProcessorLoggerDecorator.processTransaction(TransactionProcessorLoggerDecorator.java:26) 
at com.dux.gateway.core.TransactionProcessorValidatorDecorator.processTransaction(TransactionProcessorValidatorDecorator.java:24) 
at com.dux.gateway.core.PaymentGateway.execute(PaymentGateway.java:87) 
at com.dux.gateway.api.controller.TransactionController.createTransaction(TransactionController.java:86) 
at jdk.internal.reflect.GeneratedMethodAccessor562.invoke(Unknown Source) 
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.base/java.lang.reflect.Method.invoke(Method.java:566) 
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197) 
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141) 
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) 
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:894) 
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) 
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) 
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1063) 
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) 
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) 
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) 
at javax.servlet.http.HttpServlet.service(HttpServlet.java:681)

异常原因分析 #

StreamResetException: stream was reset: CANCEL 表示HTTP/2连接中的流被主动取消，结合生产环境仅在Nginx代理下出现的现象，核心问题集中在Nginx与OkHttp3的HTTP/2兼容性冲突：

1. Nginx 1.14.x的HTTP/2潜在缺陷
   Nginx 1.14.x作为早期支持HTTP/2的版本，存在流处理边界问题：
   • 当proxy_buffering off时，Nginx直接透传请求体到后端，若后端响应延迟或连接出现短暂波动，Nginx可能主动重置HTTP/2流
   • 1.14.0版本的http2_max_concurrent_streams默认值较小，高并发场景下易出现流竞争，触发流重置
2. OkHttp3 3.8.0的HTTP/2实现不足
   该版本客户端存在流状态清理不彻底的问题：
   • 长连接复用场景下，流重置后未正确释放资源，导致后续请求复用失效
   • 缺乏针对流重置场景的重试机制，单次异常直接抛出而非尝试重建连接

解决方案 #

1. Nginx配置优化
   调整HTTP/2相关参数，或临时降级协议验证问题：

   # 增大并发流数，避免流竞争
   http2_max_concurrent_streams 100;
   # 临时降级为HTTP/1.1，验证是否为HTTP/2兼容性问题
   proxy_http_version 1.1;
   proxy_set_header Connection "";
   # 开启缓冲，减少Nginx因响应延迟主动重置流的概率
   proxy_buffering on;
   

2. OkHttp3版本升级
   升级至OkHttp3 3.14.x或更高稳定版（推荐4.x），该版本修复了大量HTTP/2流处理Bug，包括流重置后的连接复用逻辑、流级别重试机制等。
3. 客户端重试逻辑补充
   在自定义HttpClient中添加针对StreamResetException的重试策略：

   public class RetryableHttpClient extends HttpClient {
       private static final int MAX_RETRY_TIMES = 3;
   
       @Override
       public String doCall(String url, String body) throws IOException {
           int retryCount = 0;
           while (retryCount < MAX_RETRY_TIMES) {
               try {
                   return super.doCall(url, body);
               } catch (StreamResetException e) {
                   retryCount++;
                   if (retryCount >= MAX_RETRY_TIMES) throw e;
                   Thread.sleep(100);
               }
           }
           throw new IOException("Max retry attempts exceeded");
       }
   }
   

4. 验证步骤
   • 先将Nginx协议降级为HTTP/1.1，观察异常是否消失，确认是否为HTTP/2兼容性问题
   • 升级OkHttp3版本后恢复HTTP/2配置，验证异常是否彻底解决

内容的提问来源于stack exchange，提问作者Oguzhan Coskun