启用CMA后调用dma_alloc_coherent()出现页分配失败求助

问题背景 #

多次运行后调用dma_alloc_coherent()出现页分配失败，系统已配置3GB CMA，每次运行前都调用dma_free_coherent()释放所有DMA内存，增大CMA容量仅延长了无问题运行次数，无法彻底解决。

内核报错日志 #

[84060.080230] Main: page allocation failure: order:10, mode:0xcc4(GFP_KERNEL|GFP_DMA32), nodemask=(null),cpuset=/,mems_allowed=0
[84060.080248] CPU: 0 PID: 7462 Comm: Main Tainted: G           OE     5.15.85evl-g0b08924b969c #3
[84060.080252] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
[84060.080253] IRQ stage: Linux
[84060.080255] Call Trace:
[84060.080257]  <TASK>
[84060.080260]  dump_stack_lvl+0x81/0xbe
[84060.080268]  dump_stack+0x10/0x16
[84060.080271]  warn_alloc.cold.144+0x7b/0xdd
[84060.080279]  ? __alloc_pages_direct_compact+0x169/0x180
[84060.080285]  __alloc_pages_slowpath.constprop.133+0xca9/0xcd0
[84060.080290]  ? inband_irq_restore+0x2b/0x40
[84060.080295]  ? inband_irq_restore+0x2b/0x40
[84060.080298]  __alloc_pages+0x2af/0x2d0
[84060.080303]  __dma_direct_alloc_pages.constprop.24+0x181/0x220
[84060.080310]  dma_direct_alloc+0x45/0x170
[84060.080314]  dma_alloc_attrs+0x39/0x60
[84060.080318]  alloc_dma_buf+0x9d/0xe0 [dma_buf_mgr]
[84060.080324]  dma_buf_mgr_ioctl+0xa9/0x280 [dma_buf_mgr]
[84060.080329]  __x64_sys_ioctl+0x91/0xd0
[84060.080334]  do_syscall_64+0x4c/0xa0
[84060.080338]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[84060.080344] RIP: 0033:0x7f4ac00d2307
[84060.080348] Code: 44 00 00 48 8b 05 69 1b 2d 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 39 1b 2d 00 f7 d8 64 89 01 48
[84060.080351] RSP: 002b:00007f4ad1e67568 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[84060.080355] RAX: ffffffffffffffda RBX: 00000000ac0a0e00 RCX: 00007f4ac00d2307
[84060.080357] RDX: 00007f4ad1e67580 RSI: 00000000c0102500 RDI: 0000000000000070
[84060.080359] RBP: 00000000002dc6c0 R08: 00007f4ad1e78700 R09: 00007f4ad1e67ac8
[84060.080361] R10: 0000000000000000 R11: 0000000000000246 R12: 00000000002dc6c0
[84060.080363] R13: 00007f4ad1e67ae8 R14: 00007f4ad1e679e0 R15: 00000000ac0b5980
[84060.080366]  </TASK>
[84060.080367] Mem-Info:
[84060.080370] active_anon:269 inactive_anon:1461195 isolated_anon:0
 active_file:526596 inactive_file:641587 isolated_file:0
 unevictable:1089001 dirty:7 writeback:0
 slab_reclaimable:74489 slab_unreclaimable:24737
 mapped:71474 shmem:4450 pagetables:9652 bounce:0
 kernel_misc_reclaimable:0
 free:88878 free_pcp:1012 free_cma:20953
[84060.080377] Node 0 active_anon:1076kB inactive_anon:5844780kB active_file:2106384kB inactive_file:2566348kB unevictable:4356004kB isolated(anon):0kB isolated(file):0kB mapped:285896kB dirty:28kB writeback:0kB shmem:17800kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 2019328kB writeback_tmp:0kB kernel_stack:7936kB pagetables:38608kB all_unreclaimable? no
[84060.080384] Node 0 DMA free:11264kB min:64kB low:80kB high:96kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15972kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[84060.080390] lowmem_reserve[]: 0 1800 15786 15786 15786
[84060.080397] Node 0 DMA32 free:90408kB min:7696kB low:9620kB high:11544kB reserved_highatomic:6144KB active_anon:64kB inactive_anon:418192kB active_file:311208kB inactive_file:8kB unevictable:968252kB writepending:8kB present:1975960kB managed:1910424kB mlocked:968252kB bounce:0kB free_pcp:520kB local_pcp:520kB free_cma:0kB
[84060.080404] lowmem_reserve[]: 0 0 13986 13986 13986
[84060.080410] Node 0 Normal free:253840kB min:59820kB low:74772kB high:89724kB reserved_highatomic:2048KB active_anon:1012kB inactive_anon:5426476kB active_file:1795176kB inactive_file:2566340kB unevictable:3387752kB writepending:20kB present:14680064kB managed:14326668kB mlocked:3387752kB bounce:0kB free_pcp:4156kB local_pcp:0kB free_cma:83812kB
[84060.080417] lowmem_reserve[]: 0 0 0 0 0
[84060.080424] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 2*4096kB (M) = 11264kB
[84060.080443] Node 0 DMA32: 36*4kB (UM) 133*8kB (UME) 309*16kB (UME) 305*32kB (UME) 197*64kB (UME) 138*128kB (UME) 67*256kB (UME) 16*512kB (UME) 1*1024kB (E) 9*2048kB (ME) 0*4096kB = 90984kB
[84060.080468] Node 0 Normal: 6742*4kB (UEC) 5324*8kB (UEHC) 3012*16kB (UMEHC) 1297*32kB (UMEHC) 415*64kB (UMEHC) 192*128kB (MEC) 1*256kB (C) 0*512kB 1*1024kB (C) 11*2048kB (C) 5*4096kB (C) = 254680kB
[84060.080495] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[84060.080497] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[84060.080500] 1201421 total pagecache pages
[84060.080501] 0 pages in swap cache
[84060.080502] Swap cache stats: add 0, delete 0, find 0/0
[84060.080504] Free swap  = 0kB
[84060.080505] Total swap = 0kB
[84060.080507] 4167999 pages RAM
[84060.080508] 0 pages HighMem/MovableOnly
[84060.080509] 104886 pages reserved
[84060.080510] 786432 pages cma reserved
[84060.080511] 0 pages hwpoisoned
[84060.080513] dma_buf_mgr ALRM:                    alloc_dma_buf: 14: Failed to allocated DMA buffer

排查思路与建议 #

1. 解决内存碎片问题 #

• 本次失败是申请order:10的连续页（即2^10=1024页，约4MB），从日志看DMA32区没有4096kB的连续空闲页，且DMA32区free_cma为0，Normal区的CMA也因碎片无法凑出连续大页。
• 每次释放DMA内存后，手动触发内存压缩：echo 1 > /proc/sys/vm/compact_memory，测试是否能恢复连续页。
• 内核启动时添加compact_unevictable_allowed=1参数，允许压缩不可回收内存来获取连续页（注意内核版本兼容性）。

2. 对齐CMA配置与DMA申请范围 #

• 报错的分配模式是GFP_DMA32，要求内存地址在0-4GB范围内，但日志中DMA32区的free_cma为0，说明CMA可能未配置在DMA32区域。
• 修改CMA启动参数为cma=3G@0-4G，强制CMA分配在DMA32范围内，确保dma_alloc_coherent()申请GFP_DMA32内存时能使用CMA。

3. 验证DMA内存释放的完整性 #

• 即使调用了dma_free_coherent()，也可能存在内存泄漏或释放不及时：
  • 用slabtop、vmstat监控每次运行前后的内存变化，确认是否有内存持续增长。
  • 检查dma_buf_mgr模块的alloc_dma_buf和释放逻辑，确认是否存在未关闭的dma_buf文件描述符，导致内存无法回收。
  • 启用内核slab调试：添加slab_debug=F启动参数，或用kmemleak工具排查泄漏点。

4. 调整内存分配策略 #

• 若业务允许，将大内存块拆分为多个小内存块分配，降低对连续大页的依赖。
• 必须使用大页时，提前用hugetlbfs预分配连续内存，避免运行时碎片问题。
• 给dma_alloc_coherent()添加GFP_COMP标志（内核版本需支持），允许通过压缩内存获取连续页。

5. 清理不可回收内存 #

• 日志中unevictable内存高达4GB+，这部分被mlock锁定的内存无法回收，挤占了连续内存空间。
• 执行cat /proc/meminfo | grep Mlocked查看锁定内存总量，排查占用进程或模块，减少不必要的内存锁定操作。

内容的提问来源于stack exchange，提问作者Russell