Java进程物理内存远超jcmd输出的原因排查

Great question! This mismatch between JVM-reported memory and actual physical memory usage is a common gotcha when working with native libraries like jnr-unixsocket. Let's break down the key reasons:

1. jnr-unixsocket's Native Memory Isn't Tracked by JVM NMT #

jnr-unixsocket relies on JNA/JNI to interact directly with the OS's Unix domain socket APIs. The memory allocated by these native operations—like kernel-side socket structures, socket receive/send buffers, and memory allocated via raw malloc() calls from the library—isn't counted by JVM's Native Memory Tracking (NMT).

NMT only tracks memory managed directly by the JVM: things like Direct ByteBuffers created via JVM APIs, JVM internal structures, and memory allocated via JNI functions that the JVM hooks into. Since jnr-unixsocket bypasses these tracked paths for many socket-related operations, that memory shows up in your OS's physical memory stats but not in jcmd/NMT output.

2. Kernel-Level Socket Buffers Add Up #

Even if you don't allocate Direct ByteBuffers yourself, every Unix domain socket connection uses kernel-allocated receive (SO_RCVBUF) and send (SO_SNDBUF) buffers. These buffers are part of your process's Resident Set Size (RSS) but are completely invisible to JVM tools.

By default, these buffers can be tens to hundreds of KB each. If your process handles a high volume of connections or sustained traffic over hours, the cumulative size of these kernel buffers can easily balloon to hundreds of MB—way beyond your JVM's configured memory limits.

3. NMT Has Blind Spots #

Even with -XX:NativeMemoryTracking=detail, NMT doesn't cover all native memory sources:

• Third-party JNI/JNA allocations: jnr-unixsocket may use JNA's memory allocation utilities, which don't notify the JVM, so NMT misses them.
• Shared libraries and loaded code: The native libraries that jnr-unixsocket depends on (or system socket libraries) are loaded into memory, contributing to RSS but not tracked as JVM-managed native memory.
• Thread stack overhead: While you set -Xss256k, OS-level thread stacks can have additional physical memory overhead that isn't fully accounted for in JVM stats.

4. CMS GC's Memory Retention (Minor Factor) #

Your CMS GC configuration helps with fragmentation, but between full compactions, the JVM might hold onto unused memory pages allocated from the OS. This can cause RSS to stay higher than the live heap size, though this is unlikely to account for the 200MB+ gap you're seeing—it's more of a minor contributor compared to native socket memory.

How to Verify & Fix This #

• Check memory maps: Use pmap -x <your-pid> to inspect your process's memory layout. Look for large anonymous mappings or entries related to socket operations to identify untracked memory sources.
• Inspect socket buffers: Run ss -m to view the memory usage of your Unix domain sockets, including their receive/send buffer sizes.
• Tune socket buffer sizes: Use jnr-unixsocket's API to explicitly set smaller SO_RCVBUF and SO_SNDBUF values for your sockets, reducing kernel memory usage.
• Ensure proper resource cleanup: Double-check that you're closing all socket connections and releasing associated resources to prevent memory leaks in the native layer.
• Update jnr-unixsocket: Make sure you're using the latest stable version—older releases may have known native memory leak issues that have been fixed.

内容的提问来源于stack exchange，提问作者Y.L.