You need to enable JavaScript to run this app.
优惠活动
大模型
产品
解决方案
定价
更多

Valgrind报告内存泄漏但无调用栈,如何定位并解决?

Valgrind检测CUDA相关内存泄漏的问题排查与解决

问题现象

使用valgrind检测内存泄漏时,报告存在明确内存泄漏,但无法显示完整调用栈,具体输出如下:

==330720== 72,704 bytes in 1 blocks are definitely lost in loss record 3,627 of 3,683
==330720==    at 0x297D90C8: malloc (vg_replace_malloc.c:381)
==330720==    by 0x4AD9433B: ???
==330720== 

已确保采用Debug构建类型,添加-g和-fno-omit-frame-pointer编译选项,同时遵循valgrind文档指导添加--caller-num参数,问题仍未解决。

排查过程

  1. 验证泄漏时机
    使用vgdb调试程序,在main函数内设置多个关键断点:
int main() {
    { 
        /* breakpoint #1 : before Load Model */
        ....
        ....

        /* breakpoint #2 : before Load Images using OpenCV */
        ...
        ...

        /* breakpoint #3 : before do inference */
        ...
        ...

        /* breakpoint #4 : before release resources */
        ...
        ...
    }
    /* breakpoint #5 : after all the resources released, 
                       ensured by the nested scope above */
    return 0;
}

所有断点处valgrind均未报告内存泄漏,仅在main()函数退出后的最终报告中出现上述72,704字节的明确泄漏。

  1. 排除全局对象因素
    编写测试代码验证全局对象的泄漏是否会被valgrind正确捕获:
class Test{
public:
    Test() { int* p = new int[10]; }
};
Test instance;
int main() {  }

valgrind能正常显示main()外的完整调用栈:

==23741== 40 bytes in 1 blocks are definitely lost in loss record 1 of 1
==23741==    at 0x4C3289F: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==23741==    by 0x1086FB: Test::Test() (in /home/***/testmem)
==23741==    by 0x1086CD: __static_initialization_and_destruction_0(int, int) (in /home/***/testmem)
==23741==    by 0x1086E3: _GLOBAL__sub_I_instance (in /home/***/testmem)
==23741==    by 0x10875C: __libc_csu_init (in /home/***/testmem)
==23741==    by 0x51E8C17: (below main) (libc-start.c:266)

由此排除全局对象导致泄漏的可能。

  1. 定位泄漏地址所属库
    查看程序链接的主要库为CUDA和OpenCV相关:
linux-vdso.so.1 (0x0000ffff94a5c000)
    libopencv_imgcodecs.so.4.1 => /home/×××/debug/samples/humanuid/../../OpenCV/lib/libopencv_imgcodecs.so.4.1 (0x0000ffff6b9a2000)
    libopencv_core.so.4.1 => /home/×××/debug/samples/humanuid/../../OpenCV/lib/libopencv_core.so.4.1 (0x0000ffff6b6eb000)
    libnvinfer.so.8 => /home/×××/debug/samples/humanuid/../../lib_static/tensorrt/lib/libnvinfer.so.8 (0x0000ffff4ef13000)
    libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000ffff4eec7000)
    libcudart.so.11.0 => /usr/local/cuda-11.4/targets/aarch64-linux/include/../lib/libcudart.so.11.0 (0x0000ffff4ee12000)
    libstdc++.so.6 => /lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000ffff4ec2d000)
    libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000ffff4eb83000)
    libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000ffff4eb5f000)
    libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffff4e9ec000)
    /lib/ld-linux-aarch64.so.1 (0x0000ffff94a2c000)
    libopencv_imgproc.so.4.1 => not found
    libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000ffff4e9d8000)
    librt.so.1 => /lib/aarch64-linux-gnu/librt.so.1 (0x0000ffff4e9c0000)
    libnvdla_compiler.so => /usr/lib/aarch64-linux-gnu/tegra/libnvdla_compiler.so (0x0000ffff4e3fc000)
    libEGL.so.1 => /lib/aarch64-linux-gnu/libEGL.so.1 (0x0000ffff4e3d8000)
    libnvmedia.so => /usr/lib/aarch64-linux-gnu/tegra/libnvmedia.so (0x0000ffff4e370000)
    libnvmedia_tensor.so => /usr/lib/aarch64-linux-gnu/tegra/libnvmedia_tensor.so (0x0000ffff4e34f000)
    libnvmedia_dla.so => /usr/lib/aarch64-linux-gnu/tegra/libnvmedia_dla.so (0x0000ffff4e336000)
    libnvos.so => /usr/lib/aarch64-linux-gnu/tegra/libnvos.so (0x0000ffff4e316000)
    libGLdispatch.so.0 => /lib/aarch64-linux-gnu/libGLdispatch.so.0 (0x0000ffff4e18b000)
    libnvvideo.so => /usr/lib/aarch64-linux-gnu/tegra/libnvvideo.so (0x0000ffff4e0b4000)
    libnvsocsys.so => /usr/lib/aarch64-linux-gnu/tegra/libnvsocsys.so (0x0000ffff4e0a0000)
    libnvrm_mem.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_mem.so (0x0000ffff4e089000)
    libnvrm_host1x.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_host1x.so (0x0000ffff4e069000)
    libnvrm_surface.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_surface.so (0x0000ffff4e02b000)
    libnvtvmr.so => /usr/lib/aarch64-linux-gnu/tegra/libnvtvmr.so (0x0000ffff4df35000)
    libnvrm_chip.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_chip.so (0x0000ffff4df21000)
    libnvrm_sync.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_sync.so (0x0000ffff4df0b000)
    libnvdc.so => /usr/lib/aarch64-linux-gnu/tegra/libnvdc.so (0x0000ffff4deec000)
    libnvparser.so => /usr/lib/aarch64-linux-gnu/tegra/libnvparser.so (0x0000ffff4dea3000)
    libnvdla_runtime.so => /usr/lib/aarch64-linux-gnu/tegra/libnvdla_runtime.so (0x0000ffff4de1c000)
    libnvrm_stream.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_stream.so (0x0000ffff4de02000)
    libnvsciipc.so => /usr/lib/aarch64-linux-gnu/tegra/libnvsciipc.so (0x0000ffff4dde0000)
    libnvimp.so => /usr/lib/aarch64-linux-gnu/tegra/libnvimp.so (0x0000ffff4ddcb000)

在泄漏调用栈中的地址0x4AD9433B设置断点,GDB显示该地址属于动态加载后卸载的共享库/usr/local/cuda/targets/aarch64-linux/lib/libnvrtc.so:

(gdb) break *0x4AD9433B
Breakpoint 1 at 0x4ad9433b
(gdb) c
Continuing.
warning: Temporarily disabling breakpoints for unloaded shared library "/usr/local/cuda/targets/aarch64-linux/lib/libnvrtc.so"
[Inferior 1 (Remote target) exited normally]

原因分析

该泄漏源于标准C++库的紧急内存池:

  • 当库被静态链接时,紧急内存池会在程序退出时由系统释放,valgrind会将其标记为“仍可达”;
  • 当库通过dlopen动态加载后再用dlclose卸载时,该内存池会变为不可访问,被valgrind判定为明确泄漏。

解决方案

静态链接包含该紧急内存池的库(本场景中为libnvrtc.so),此时紧急内存池始终处于可达状态,不会被valgrind判定为明确泄漏。


内容的提问来源于stack exchange,提问作者Melina

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级,最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分), 限时 9.9 元起

最近更新时间:2026.08.24 21:03:29