WSO2 API Manager 3.2.0频繁出现OOM错误，求排查原因

我们在使用WSO2 API Manager 3.2.0时频繁遇到OOM错误，该错误并非堆空间不足导致，而是运行时尝试创建大小为INTEGER.MAX_VALUE的byte[]时触发。堆转储中的错误堆栈如下：

"PassThroughMessageProcessor-301" prio=5 tid=2866 RUNNABLE
at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:48)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:239)
   local variable: org.apache.synapse.commons.json.JsonUtil$ReadOnlyBIS#1
   local variable: byte[]#859071
at java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
   local variable: org.apache.synapse.commons.json.JsonUtil$ReadOnlyBIS#1
   local variable: byte[]#680420
at java.io.BufferedInputStream.read(BufferedInputStream.java:351)
   local variable: org.apache.synapse.commons.json.JsonUtil$ReadOnlyBIS#1
   local variable: byte[]#680420
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792)
   local variable: org.apache.synapse.commons.json.JsonUtil$ReadOnlyBIS#1
   local variable: org.apache.synapse.transport.passthru.Pipe$ByteBufferOutputStream#1
   local variable: byte[]#680420
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)
at org.apache.synapse.commons.json.JsonUtil.writeJsonStream(JsonUtil.java:1168)
   local variable: org.apache.synapse.commons.json.JsonUtil$ReadOnlyBIS#1
   local variable: org.apache.axis2.context.MessageContext#50
at org.apache.synapse.commons.json.JsonUtil.writeAsJson(JsonUtil.java:378)
   local variable: org.apache.synapse.commons.json.JsonUtil$ReadOnlyBIS#1
at org.apache.synapse.commons.json.JsonStreamFormatter.writeTo(JsonStreamFormatter.java:70)
   local variable: org.apache.axis2.context.MessageContext#50
at org.apache.synapse.transport.passthru.PassThroughHttpSender.sendRequestContent(PassThroughHttpSender.java:414)
   local variable: org.apache.synapse.transport.passthru.PassThroughHttpSSLSender#1
   local variable: org.apache.synapse.transport.passthru.Pipe#23
at org.apache.synapse.transport.passthru.PassThroughHttpSender.invoke(PassThroughHttpSender.java:277)
   local variable: org.apache.axis2.context.MessageContext#50
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
   local variable: org.apache.axis2.context.MessageContext#50
   local variable: com.dynatrace.agent.TraceTag#868
at org.apache.synapse.core.axis2.DynamicAxisOperation$DynamicOperationClient.send(DynamicAxisOperation.java:185)
   local variable: org.apache.synapse.core.axis2.DynamicAxisOperation$DynamicOperationClient#1
   local variable: org.apache.axis2.context.MessageContext#50
   local variable: org.apache.axis2.context.MessageContext#51
at org.apache.synapse.core.axis2.DynamicAxisOperation$DynamicOperationClient.executeImpl(DynamicAxisOperation.java:167)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
at org.apache.synapse.core.axis2.Axis2FlexibleMEPClient.send(Axis2FlexibleMEPClient.java:656)
at org.apache.synapse.core.axis2.Axis2Sender.sendOn(Axis2Sender.java:86)
   local variable: org.apache.synapse.core.axis2.Axis2MessageContext#5
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:573)
at org.apache.synapse.endpoints.AbstractEndpoint.send(AbstractEndpoint.java:409)
   local variable: org.apache.synapse.endpoints.HTTPEndpoint#97
   local variable: org.apache.synapse.core.axis2.Axis2MessageContext#5
at org.apache.synapse.endpoints.HTTPEndpoint.send(HTTPEndpoint.java:100)
at org.apache.synapse.endpoints.IndirectEndpoint.send(IndirectEndpoint.java:56)
at org.apache.synapse.mediators.builtin.SendMediator.mediate(SendMediator.java:123)
   local variable: org.apache.synapse.mediators.MediatorLog#1
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:109)
   local variable: org.apache.synapse.config.xml.AnonymousListMediator#187
   local variable: org.apache.synapse.core.axis2.Axis2MessageContext#5
   local variable: org.apache.synapse.mediators.MediatorLog#2
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:71)
at org.apache.synapse.config.xml.AnonymousListMediator.mediate(AnonymousListMediator.java:37)
at org.apache.synapse.mediators.filters.FilterMediator.mediate(FilterMediator.java:205)
   local variable: org.apache.synapse.core.axis2.Axis2MessageContext#5
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:109)
   local variable: org.apache.synapse.mediators.base.SequenceMediator#403
   local variable: org.apache.synapse.core.axis2.Axis2MessageContext#5
   local variable: org.apache.synapse.mediators.MediatorLog#3
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:71)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:158)
   local variable: org.apache.synapse.mediators.base.SequenceMediator#403
   local variable: org.apache.synapse.core.axis2.Axis2MessageContext#5
   local variable: org.apache.synapse.mediators.MediatorLog#4
at org.apache.synapse.rest.Resource.process(Resource.java:331)
   local variable: org.apache.synapse.rest.Resource#186
   local variable: org.apache.synapse.core.axis2.Axis2MessageContext#5
at org.apache.synapse.rest.API.process(API.java:447)
at org.apache.synapse.rest.RESTRequestHandler.apiProcessNonDefaultStrategy(RESTRequestHandler.java:144)
at org.apache.synapse.rest.RESTRequestHandler.identifyAPI(RESTRequestHandler.java:164)
at org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:95)
   local variable: org.apache.synapse.rest.RESTRequestHandler#1
   local variable: org.apache.synapse.core.axis2.Axis2MessageContext#5
   local variable: java.util.ArrayList#139741
at org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:73)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:331)
   local variable: org.apache.synapse.core.axis2.Axis2MessageContext#5
at org.apache.synapse.core.axis2.SynapseMessageReceiver.receive(SynapseMessageReceiver.java:99)
   local variable: org.apache.synapse.core.axis2.SynapseMessageReceiver#1
   local variable: org.apache.synapse.core.axis2.Axis2MessageContext#5
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
   local variable: org.apache.axis2.context.MessageContext#52
at org.apache.synapse.transport.passthru.ServerWorker.processNonEntityEnclosingRESTHandler(ServerWorker.java:367)
   local variable: org.apache.synapse.transport.passthru.ServerWorker#18
at org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:426)
   local variable: org.apache.synapse.transport.passthru.ServerWorker#18
   local variable: com.dynatrace.agent.TraceTag#868
at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:181)
   local variable: org.apache.synapse.transport.passthru.ServerWorker#18
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
   local variable: org.apache.axis2.transport.base.threads.NativeWorkerPool$1#6
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   local variable: org.apache.axis2.transport.base.threads.Axis2ThreadPoolExecutor#1
   local variable: java.util.concurrent.ThreadPoolExecutor$Worker#908
   local variable: java.lang.Thread#981
   local variable: org.apache.axis2.transport.base.threads.NativeWorkerPool$1#6
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:834)

触发OOM的BufferedInputStream.java第239行代码：

byte[] nbuf = new byte[nsz]; 

可能的原因分析 #

• JsonUtil$ReadOnlyBIS的available()方法返回异常值
  BufferedInputStream.fill()中用于创建数组的nsz值，是基于底层流的available()方法计算而来。如果ReadOnlyBIS的available()实现存在逻辑错误，比如在处理异常流（如截断的JSON、未正确关闭的流）时，返回了Integer.MAX_VALUE，就会直接触发创建超大数组的操作。结合堆栈看，这个流是从Pipe$ByteBufferOutputStream拷贝的，若Pipe内部缓冲区状态异常，也会导致available()返回错误数值。

• 请求/响应的Content-Length头部异常
  如果上游请求或后端响应的Content-Length被错误设置为Integer.MAX_VALUE，Synapse的JSON处理逻辑可能会根据这个头部值来计算需要读取的字节数，进而传递给BufferedInputStream，导致尝试创建对应大小的数组。这种情况通常出现在后端服务返回错误头部，或者API网关在转发请求时头部处理逻辑出错。

• Synapse JSON处理的边界逻辑Bug
  在JsonUtil.writeJsonStream()方法处理特殊场景时，比如空JSON payload、格式错误的JSON、超大JSON数据，可能存在整数溢出或逻辑漏洞，导致计算需要读取的字节数时出现异常，最终得到Integer.MAX_VALUE。例如，当计算字节数时的变量溢出，原本的小数值被错误转换为最大值。

• 高并发场景下的Pipe资源异常
  在高流量并发场景中，PassThroughHttpSender使用的Pipe资源若未正确释放或复用，可能导致内部的ByteBuffer状态错乱，使得available()读取到错误的数值，进而触发错误的数组大小计算。

内容的提问来源于stack exchange，提问作者Anshul Sood