嵌入Wildfly 31的Camunda JVM直接内存泄漏问题求助

我们有一款基于Wildfly嵌入Camunda Engine的业务应用，版本信息如下：

• Camunda 7.21
• Wildfly 31

应用在BPMN流程中通过Camunda HTTP Connector读取大量数据，该连接器内部使用Apache HTTP Client。运行一周后Pod因OOM重启，经多次测试和Native Memory Tracking分析，发现问题源于Undertow字节缓冲池的直接内存使用——该内存以每日300MB的速度增长，最终导致OOM。

以下是运行一天后的Native Memory Tracking差异数据：

Native Memory Tracking:

(Omitting categories weighting less than 1MB)

Total: reserved=3383MB +221MB, committed=1330MB +669MB

-                 Java Heap (reserved=2152MB, committed=658MB +400MB)
                            (mmap: reserved=2152MB, committed=658MB +400MB)
 
-                     Class (reserved=5MB, committed=5MB)
                            (classes #28128 +393)
                            (  instance classes #26459 +370, array classes #1669 +23)
                            (malloc=5MB #114630 +7518)
                           : (  Metadata)
                            (    reserved=192MB, committed=182MB +4MB)
                            (    used=179MB +4MB)
                            (    waste=3MB =1.69% +1MB)
                           : (  Class space)
                            (    reserved=208MB, committed=22MB)
                            (    used=20MB)
                            (    waste=2MB =10.09%)
 
-                    Thread (reserved=157MB -22MB, committed=13MB -9MB)
                            (thread #0)
                            (stack: reserved=157MB -22MB, committed=13MB -9MB)
 
-                      Code (reserved=249MB +1MB, committed=85MB +17MB)
                            (malloc=7MB +1MB #25344 +3034)
                            (mmap: reserved=242MB, committed=79MB +15MB)
 
-                        GC (reserved=124MB +3MB, committed=69MB +17MB)
                            (malloc=12MB +3MB #42350 +8980)
                            (mmap: reserved=112MB, committed=57MB +15MB)
 
-                  Compiler (reserved=1MB -1MB, committed=1MB -1MB)
                            (malloc=1MB -1MB #1944 -297)
 
-                  Internal (reserved=1MB, committed=1MB)
                            (malloc=1MB #33414 +3969)
 
                     Other (reserved=246MB +241MB, committed=246MB +241MB)
                            (malloc=246MB +241MB #15742 +15318)***
 
-                    Symbol (reserved=28MB, committed=28MB)
                            (malloc=25MB #706302 +10200)
                            (arena=3MB #1)
 
-           Native Memory Tracking (reserved=16MB +1MB, committed=16MB +1MB)
                            (malloc=1MB #10560 +4262)
                            (tracking overhead=15MB +1MB)
 
-               Arena Chunk (reserved=0MB -2MB, committed=0MB -2MB)
                            (malloc=0MB -2MB)
 
-                    Module (reserved=1MB, committed=1MB)
                            (malloc=1MB #6347 -24)
 
-                 Metaspace (reserved=194MB, committed=184MB +5MB)
                            (malloc=2MB #1544 +88)
                            (mmap: reserved=192MB, committed=182MB +4MB)
 
-                   Unknown (reserved=208MB, committed=22MB)
                            (mmap: reserved=208MB, committed=22MB)

可以看到Other部分增长了241MB，其他内存区域（如堆内存）均为正常波动，怀疑Undertow字节缓冲池存在内存泄漏。

1. 配置Undertow字节缓冲池参数 #

执行以下Wildfly CLI命令：

/subsystem=undertow/byte-buffer-pool=default:write-attribute(name=buffer-size,value=16384)
/subsystem=undertow/byte-buffer-pool=default:write-attribute(name=leak-detection-percent,value=70)
/subsystem=undertow/byte-buffer-pool=default:write-attribute(name=max-pool-size,value=20)

效果：无任何改善。

2. 通过JVM参数限制直接内存使用 #

添加以下JVM启动参数：

-XX:MaxDirectMemorySize=256M -Djdk.nio.maxCachedBufferSize=16384
效果：应用抛出直接内存分配失败错误：

Caused by: java.lang.OutOfMemoryError: Cannot reserve 16384 bytes of direct buffer memory (allocated: 805298186, limit: 805306368)
    at java.base/java.nio.Bits.reserveMemory(Unknown Source)
    at java.base/java.nio.DirectByteBuffer.<init>(Unknown Source)
    at java.base/java.nio.ByteBuffer.allocateDirect(Unknown Source)

3. 禁用Undertow字节缓冲池的直接内存使用 #

执行以下Wildfly CLI命令：

/subsystem=undertow/byte-buffer-pool=default:write-attribute(name=direct,value=false)

效果：RSS内存缓存持续增长，相关top命令输出如下：

top - 05:46:29 up 101 days, 18:56,  0 users,  load average: 1.53, 0.99, 0.96
Tasks:  15 total,   1 running,  14 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.6 us,  0.6 sy,  0.0 ni, 97.3 id,  0.0 wa,  0.1 hi,  0.2 si,  0.2 st
MiB Mem :  64292.6 total,   3543.6 free,  11899.7 used,  48849.2 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  50339.2 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                               
   1596 501       20   0 3367.7m   2.7g  33.7m S   0.7   4.2 491:38.37 java                                                                 
      1 501       20   0   11.8m   3.0m   2.6m S   0.0   0.0   0:00.03 entrypoint.sh                                                         
     48 501       20   0   12.3m   3.6m   2.6m S   0.0   0.0   0:00.02 docker.wildfly.                                                       
   1392 501       20   0   12.3m   3.5m   2.6m S   0.0   0.0   0:00.00 service.sh                                                            
   1402 501       20   0   12.3m   3.5m   2.5m S   0.0   0.0   0:00.00 standalone.sh  

目前对该问题的解决毫无头绪，恳请技术支持。

内容的提问来源于stack exchange，提问作者Lucia