Docker因OOM杀死Java容器,但JMX无高内存占用迹象求助
AWS ECS Java 11容器内存异常排查求助
环境与配置
我们在AWS ECS上运行Java 11应用的Docker容器,服务配置如下,当内存占用达1.5GB时容器会被终止:
"memory": 1500, "memoryReservation": 1137,
AWS ECS服务指标显示容器因触发内存阈值被终止
监控异常现象
监控仪表盘中未观测到内存占用增长迹象(注:Grafana仪表盘中的水平线因Prometheus停止接收指标导致)
Java内存限制配置
已通过JVM参数严格限制堆内及堆外内存使用,启动命令如下:
java ‑Xmx600m ‑XX:+UseG1GC ‑XX:MaxMetaspaceSize=220m ‑XX:+PrintFlagsFinal ‑XX:NewRatio=1 ‑XX:MaxDirectMemorySize=40m ‑Xss512k ‑XX:ProfiledCodeHeapSize=70M ‑XX:NonProfiledCodeHeapSize=50M ‑XX:NonNMethodCodeHeapSize=8M ‑XX:+UseCodeCacheFlushing ‑XX:CompressedClassSpaceSize=32M ‑XX:ReservedCodeCacheSize=128M ‑XX:+SegmentedCodeCache ‑javaagent:/opentelemetry‑javaagent.jar ‑Dotel.javaagent.extensions=/opentelemetry‑java‑ecs‑extension‑all.jar ‑Dotel.resource.attributes=xxx,aws.ecs.launchtype=ec2 ‑XX:NativeMemoryTracking=summary ‑XX:‑OmitStackTraceInFastThrow ‑Dspring.profiles.active=prd‑ecs ‑Dspring.cloud.consul.host=172.17.42.1 ‑Dspring.cloud.consul.config.watch.enabled=false ‑Dcom.sun.management.jmxremote.port=9999 ‑Dcom.sun.management.jmxremote.authenticate=false ‑Dcom.sun.management.jmxremote.ssl=false ‑Dlog4j2.formatMsgNoLookups=true ‑Dserver.port=8080 ‑Daws.paramstore.enabled=true ‑Daws.paramstore.region=eu‑west‑1 ‑Dspring.main.banner‑mode=off ‑jar /1234‑service.jar
Native Memory Tracking(NMT)结果
运行最久的容器NMT输出显示,JVM总预留内存仅约1.1GB:
Native Memory Tracking: Total: reserved=1180115KB, committed=700171KB - Java Heap (reserved=614400KB, committed=260096KB) (mmap: reserved=614400KB, committed=260096KB) - Class (reserved=215344KB, committed=208080KB) (classes #35567) ( instance classes #33401, array classes #2166) (malloc=8496KB #125896) (mmap: reserved=206848KB, committed=199584KB) ( Metadata: ) ( reserved=174080KB, committed=173536KB) ( used=167363KB) ( free=6173KB) ( waste=0KB =0.00%) ( Class space:) ( reserved=32768KB, committed=26048KB) ( used=22342KB) ( free=3706KB) ( waste=0KB =0.00%) - Thread (reserved=77858KB, committed=16034KB) (thread #125) (stack: reserved=77260KB, committed=15436KB) (malloc=452KB #752) (arena=146KB #249) - Code (reserved=139426KB, committed=96022KB) (malloc=7330KB #29103) (mmap: reserved=132096KB, committed=88692KB) - GC (reserved=70010KB, committed=56862KB) (malloc=14362KB #61984) (mmap: reserved=55648KB, committed=42500KB) - Compiler (reserved=896KB, committed=896KB) (malloc=764KB #2798) (arena=133KB #5) - Internal (reserved=4509KB, committed=4509KB) (malloc=4477KB #3670) (mmap: reserved=32KB, committed=32KB) - Other (reserved=4359KB, committed=4359KB) (malloc=4359KB #39) - Symbol (reserved=37775KB, committed=37775KB) (malloc=34219KB #439593) (arena=3556KB #1) - Native Memory Tracking (reserved=10765KB, committed=10765KB) (malloc=34KB #437) (tracking overhead=10731KB) - Arena Chunk (reserved=190KB, committed=190KB) (malloc=190KB) - Logging (reserved=4KB, committed=4KB) (malloc=4KB #194) - Arguments (reserved=20KB, committed=20KB) (malloc=20KB #545) - Module (reserved=3692KB, committed=3692KB) (malloc=3692KB #14082) - Synchronizer (reserved=860KB, committed=860KB) (malloc=860KB #7286) - Safepoint (reserved=8KB, committed=8KB) (mmap: reserved=8KB, committed=8KB)
Docker镜像配置
90+个Docker服务共用的镜像定义:
FROM --platform=$TARGETPLATFORM amazoncorretto:11 RUN yum install -y shadow-utils wget && yum clean all RUN wget https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar -O /opentelemetry-javaagent.jar ADD opentelemetry-java-ecs-extension-*-all.jar /opentelemetry-java-ecs-extension-all.jar RUN adduser docker RUN if [ -f $JAVA_HOME/conf/security/java.security ]; then sed -i 's/^networkaddress.cache.ttl/#networkaddress.cache.ttl/g' $JAVA_HOME/conf/security/java.security; fi && \ echo "networkaddress.cache.ttl=30" >> $JAVA_HOME/conf/security/java.security USER docker
环境变量配置
已设置以下环境变量:
{ "name": "MALLOC_ARENA_MAX", "value": "4" }
核心问题
容器因内存占用达1.5GB被终止,但NMT显示JVM总预留内存仅1.1GB,且JMX指标未观测到内存增长,无法解释额外400MB内存的来源,恳请协助排查。
内容的提问来源于Stack Exchange,提问作者Phillip
相关产品推荐
相关产品推荐

