slab_unreclaimable内存过高引发OOM杀死进程,请求排查内存泄漏进程
slab_unreclaimable内存过高引发OOM杀死进程,请求排查内存泄漏进程
你好,从你提供的内核日志来看,slab_unreclaimable:373525确实占用了大量内核内存,这大概率是导致OOM杀死robot_sensing_s进程的核心原因。别着急,我给你几个实用的排查步骤,帮你定位问题根源:
- 先用
slabtop工具实时监控slab内存的分布情况,重点关注NAME列下各个slab缓存的占用比例和增长趋势,找出占用内存最高的几个缓存项。你可以用slabtop -o > slab_snapshot.txt导出快照,方便后续对比分析。 - 针对找到的高占用slab缓存,通过
find /sys/kernel/slab -name "<目标缓存名>"定位到对应的sysfs目录,查看目录下的allocators文件,这里会记录分配该slab的内核函数,能帮你关联到具体的内核模块或者功能。 - 如果想进一步关联到用户进程,可以用
perf工具追踪slab分配事件:执行perf record -g -e kmalloc后,复现你的业务场景,再用perf report查看调用栈,就能找到频繁触发slab分配的进程和内核路径。 - 另外,有些用户进程的异常操作(比如频繁创建销毁文件句柄、网络连接)也会引发slab增长,你可以用
ps aux --sort=-%mem查看用户进程的内存占用情况,排查是否有进程存在异常行为。 - 最后,你可以定期查看
/proc/meminfo里的Slab、SReclaimable、SUnreclaim字段,确认不可回收slab是持续增长的,排除临时峰值的可能性。
以下是你提供的内核日志:
Oct 29 10:34:34 kernel: [43384.171352] Mem-Info: Oct 29 10:34:34 kernel: [43384.171357] active_anon:514088 inactive_anon:280946 isolated_anon:0 Oct 29 10:34:34 kernel: [43384.171357] active_file:247 inactive_file:1134 isolated_file:123 Oct 29 10:34:34 kernel: [43384.171357] unevictable:31537 dirty:0 writeback:0 unstable:0 Oct 29 10:34:34 kernel: [43384.171357] slab_reclaimable:18057 slab_unreclaimable:373525 Oct 29 10:34:34 kernel: [43384.171357] mapped:19912 shmem:59183 pagetables:742629 bounce:0 Oct 29 10:34:34 kernel: [43384.171357] free:26336 free_pcp:1351 free_cma:0 Oct 29 10:34:34 kernel: [43384.171361] Node 0 active_anon:2056352kB inactive_anon:1123784kB active_file:988kB inactive_file:4536kB unevictable:126148kB isolated(anon):0kB isolated(file):492kB mapped:79648kB dirty:0kB writeback:0kB shmem:236732kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no Oct 29 10:34:34 kernel: [43384.171363] Node 0 DMA free:15888kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15888kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Oct 29 10:34:34 kernel: [43384.171368] lowmem_reserve[]: 0 2116 7795 7795 7795 Oct 29 10:34:34 kernel: [43384.171370] Node 0 DMA32 free:40392kB min:18316kB low:22892kB high:27468kB active_anon:297972kB inactive_anon:260848kB active_file:108kB inactive_file:1248kB unevictable:33480kB writepending:0kB present:2275704kB managed:2210016kB mlocked:32256kB kernel_stack:260kB pagetables:1060844kB bounce:0kB free_pcp:912kB local_pcp:224kB free_cma:0kB Oct 29 10:34:34 kernel: [43384.171374] lowmem_reserve[]: 0 0 5678 5678 5678 Oct 29 10:34:34 kernel: [43384.171376] Node 0 Normal free:49064kB min:49132kB low:61412kB high:73692kB active_anon:1758380kB inactive_anon:862936kB active_file:1160kB inactive_file:4708kB unevictable:92668kB writepending:0kB present:5996544kB managed:5822328kB mlocked:49888kB kernel_stack:24860kB pagetables:1909672kB bounce:0kB free_pcp:4492kB local_pcp:856kB free_cma:0kB Oct 29 10:34:34 kernel: [43384.171381] lowmem_reserve[]: 0 0 0 0 0 Oct 29 10:34:34 kernel: [43384.171383] Node 0 DMA: 2*4kB (U) 1*8kB (U) 2*16kB (U) 1*32kB (U) 3*64kB (U) 2*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15888kB Oct 29 10:34:34 kernel: [43384.171390] Node 0 DMA32: 763*4kB (UME) 298*8kB (UME) 870*16kB (UME) 100*32kB (UME) 59*64kB (UE) 37*128kB (UE) 24*256kB (UE) 8*512kB (UE) 0*1024kB 0*2048kB 0*4096kB = 41308kB Oct 29 10:34:34 kernel: [43384.171395] Node 0 Normal: 2145*4kB (UME) 2366*8kB (UME) 834*16kB (UME) 144*32kB (UME) 40*64kB (UME) 9*128kB (UME) 2*256kB (UE) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 49684kB Oct 29 10:34:34 kernel: [43384.171402] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Oct 29 10:34:34 kernel: [43384.171403] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Oct 29 10:34:34 kernel: [43384.171403] 84105 total pagecache pages Oct 29 10:34:34 kernel: [43384.171405] 22211 pages in swap cache Oct 29 10:34:34 kernel: [43384.171406] Swap cache stats: add 734033, delete 711823, find 375430/426343 Oct 29 10:34:34 kernel: [43384.171407] Free swap = -1123492kB Oct 29 10:34:34 kernel: [43384.171407] Total swap = 0kB Oct 29 10:34:34 kernel: [43384.171408] 2072060 pages RAM Oct 29 10:34:34 kernel: [43384.171408] 0 pages HighMem/MovableOnly Oct 29 10:34:34 kernel: [43384.171409] 60002 pages reserved Oct 29 10:34:34 kernel: [43384.171409] 0 pages cma reserved ... Out of memory: Killed process 4418 (robot_sensing_s) total-vm:644708848kB, anon-rss:264464kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:1257132kB oom_score_adj:0
备注:内容来源于stack exchange,提问作者GLEN
相关产品推荐
相关产品推荐

