K8s(EKS)中Go服务RES内存过高被OOMKilled，是否无内存泄漏？

在AWS EKS的Kubernetes集群中运行Go服务时，发现进程RES内存增长至Pod的resources.limits.memory上限365Mi时被Kubernetes OOMKilled。即使将Pod内存上限调至2GB，仍会出现相同情况。GC日志显示服务内存占用很小，但内存并未完全释放给操作系统。

K8s配置与环境变量 #

测试时Pod内存配置为365Mi（2GB时情况一致），K8s资源清单配置如下：

resources:
  limits:
    memory: 365Mi
  requests:
    cpu: 1000m
    memory: 365Mi

环境变量配置：

GOMEMLIMIT: '180MiB'
GOMAXPROCS: '2'
GODEBUG: 'gctrace=1'
GOGC: '50'

最初未设置GOMEMLIMIT，查阅资料后设为Pod内存上限的一半，无改善；将默认GOGC改为50，也无效果。

垃圾回收（GC）日志 #

进程被杀死前的GC日志：

21:03:41.854 gc 1643 @80939.735s 0%: 0.060+2.4+0.002 ms clock, 0.12+1.1/1.4/0.57+0.005 ms cpu, 12->12->8 MB, 12 MB goal, 0 MB stacks, 0 MB globals, 2 P

Memstats数据 #

手动转换为MB的runtime.mstats数据：

21:03:43.195 
{
  "Alloc":8922888 (8.9MB),
  "TotalAlloc":5646312096 (5.6GB),
  "Sys":28415240 (28.4MB),
  "HeapSys":18284544 (18.2MB),
  "HeapIdle":6414336 (6.4MB),
  "HeapReleased":3121152 (3.1MB),
  "HeapInuse":11870208 (11.8MB),
  "HeapObjects":24393,
  "MallocsObjects":43016155,
  "FreesObjects":42991762,
  "LiveObjects":24393,
  "PauseTotalNs":153890330,
  "NumGC":1643,
  "NumGoroutine":265
}

Alloc为8.9MB，与GC日志中的最终值8MB（12->12->8 MB）一致。

OOMKilled前后的日志样本：

15:39:21.969  my-service gc 1709 @168600.017s 0%: 0.033+3.5+0.002 ms clock, 0.033+0/0.059/3.4+0.002 ms cpu, 12->12->9 MB, 19 MB goal, 0 MB stacks, 0 MB globals, 1 P (forced)
15:39:23.947  my-service {"Alloc":10126368,"TotalAlloc":5661878296,"Sys":36803848,"HeapSys":26771456,"HeapIdle":13369344,"HeapReleased":13336576,"HeapObjects":42613,"MallocsObjects":35141353,"FreesObjects":35098740,"LiveObjects":42613,"PauseTotalNs":70123823,"NumGC":1709,"NumGoroutine":264}
15:40:23.948  my-service {"Alloc":14120360,"TotalAlloc":5665872288,"Sys":36803848,"HeapSys":26738688,"HeapIdle":10780672,"HeapReleased":10780672,"HeapObjects":73826,"MallocsObjects":35172566,"FreesObjects":35098740,"LiveObjects":73826,"PauseTotalNs":70123823,"NumGC":1709,"NumGoroutine":264}
15:41:16.861  my-service Killed
15:41:18.201  my-service gc 1 @0.015s 6%: 0.007+4.9+0.002 ms clock, 0.007+0.027/1.3/0+0.002 ms cpu, 3->4->2 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 1 P

top命令输出 #

kubectl top pod输出：

NAME                              CPU(cores)   MEMORY(bytes)
my-service-pod-56f7fcffbb-d8tdh   8m           344Mi

容器内top命令输出：

top - 05:04:05 up 14 days,  8:05,  0 user,  load average: 1.78, 1.95, 1.89
Tasks:   4 total,   1 running,   3 sleeping,   0 stopped,   0 zombie
%Cpu(s): 13.2 us,  5.2 sy,  0.0 ni, 79.2 id,  0.8 wa,  0.0 hi,  1.6 si,  0.0 st
MiB Mem :  15801.5 total,   4087.7 free,   9661.0 used,   2460.5 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   6140.5 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
     11 guest     20   0 2315940 348884  10532 S   0.7   2.2   7:02.35 my-service

pprof分析 #

pprof未发现异常，总分配内存为7096.39kB：

go tool pprof ~/Downloads/my-service-pod-56f7fcffbb-d8tdh.tar.gz
File: my-service
Type: inuse_space
Time: Feb 6, 2024 at 8:40pm (PST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 7096.39kB, 100% of 7096.39kB total
Showing top 10 nodes out of 35
      flat  flat%   sum%        cum   cum%
    4104kB 57.83% 57.83%     4104kB 57.83%  github.com/DataDog/datadog-go/v5/statsd.newStatsdBuffer (inline)
  902.59kB 12.72% 70.55%   902.59kB 12.72%  compress/flate.NewWriter (inline)
  553.04kB  7.79% 78.34%   553.04kB  7.79%  gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer.newConcentrator
  512.34kB  7.22% 85.56%   512.34kB  7.22%  crypto/x509.map.init.0
  512.31kB  7.22% 92.78%   512.31kB  7.22%  vendor/golang.org/x/net/http/httpguts.map.init.0
  512.10kB  7.22%   100%   512.10kB  7.22%  github.com/aws/aws-sdk-go/aws/endpoints.init
         0     0%   100%   902.59kB 12.72%  bufio.(*Writer).Flush
         0     0%   100%   902.59kB 12.72%  compress/gzip.(*Writer).Write
         0     0%   100%   512.34kB  7.22%  crypto/x509.init
         0     0%   100%     4104kB 57.83%  github.com/DataDog/datadog-go/v5/statsd.(*bufferPool).addNewBuffer
(pprof)

问题判断与疑问 #

可以基于以下几点判断服务不存在内存泄漏：

• runtime.MemStats.Sys数值较低；
• runtime.Memstats.NumGC保持稳定，协程数量无异常增长；
• runtime.Memstats.TotalAlloc数值大是因为它是服务启动以来的累计分配量，大部分内存已被释放；
• 每次GC仅需释放少量内存，GC目标内存从未超过18MB。

GC确实在释放内存，但并非所有内存都归还给了操作系统，服务始终占用resources.limits.memory设定的最大内存。

解决建议 #

• 强制内存归还给OS：尽管Go 1.21+默认使用MADV_FREE，但内核可能不会及时回收空闲内存。可显式设置GODEBUG=madvdontneed=1，强制使用MADV_DONTNEED策略，让Go将空闲内存立即归还给操作系统，此操作可能有轻微性能损耗，但能缓解OOM问题。
• 排查Cgo依赖：如果服务使用了Cgo或第三方C库，这类内存不受Go GC管理，可能导致RSS居高不下。需要检查此类依赖的内存分配与释放逻辑。
• 调整GOMEMLIMIT：当前设置为180Mi，可尝试调至更接近Pod上限但留有余量的值（如300Mi），让Go GC更积极地触发回收，避免堆内存过度扩张。
• 优化GC触发频率：降低GOGC值（如设为30），让GC更频繁运行，促进空闲内存释放给操作系统。

内容的提问来源于stack exchange，提问作者Miguel Reyes