Valgrind报告内存泄漏但无调用栈,如何定位并解决?
Valgrind检测CUDA相关内存泄漏的问题排查与解决
问题现象
使用valgrind检测内存泄漏时,报告存在明确内存泄漏,但无法显示完整调用栈,具体输出如下:
==330720== 72,704 bytes in 1 blocks are definitely lost in loss record 3,627 of 3,683 ==330720== at 0x297D90C8: malloc (vg_replace_malloc.c:381) ==330720== by 0x4AD9433B: ??? ==330720==
已确保采用Debug构建类型,添加-g和-fno-omit-frame-pointer编译选项,同时遵循valgrind文档指导添加--caller-num参数,问题仍未解决。
排查过程
- 验证泄漏时机
使用vgdb调试程序,在main函数内设置多个关键断点:
int main() { { /* breakpoint #1 : before Load Model */ .... .... /* breakpoint #2 : before Load Images using OpenCV */ ... ... /* breakpoint #3 : before do inference */ ... ... /* breakpoint #4 : before release resources */ ... ... } /* breakpoint #5 : after all the resources released, ensured by the nested scope above */ return 0; }
所有断点处valgrind均未报告内存泄漏,仅在main()函数退出后的最终报告中出现上述72,704字节的明确泄漏。
- 排除全局对象因素
编写测试代码验证全局对象的泄漏是否会被valgrind正确捕获:
class Test{ public: Test() { int* p = new int[10]; } }; Test instance; int main() { }
valgrind能正常显示main()外的完整调用栈:
==23741== 40 bytes in 1 blocks are definitely lost in loss record 1 of 1 ==23741== at 0x4C3289F: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==23741== by 0x1086FB: Test::Test() (in /home/***/testmem) ==23741== by 0x1086CD: __static_initialization_and_destruction_0(int, int) (in /home/***/testmem) ==23741== by 0x1086E3: _GLOBAL__sub_I_instance (in /home/***/testmem) ==23741== by 0x10875C: __libc_csu_init (in /home/***/testmem) ==23741== by 0x51E8C17: (below main) (libc-start.c:266)
由此排除全局对象导致泄漏的可能。
- 定位泄漏地址所属库
查看程序链接的主要库为CUDA和OpenCV相关:
linux-vdso.so.1 (0x0000ffff94a5c000) libopencv_imgcodecs.so.4.1 => /home/×××/debug/samples/humanuid/../../OpenCV/lib/libopencv_imgcodecs.so.4.1 (0x0000ffff6b9a2000) libopencv_core.so.4.1 => /home/×××/debug/samples/humanuid/../../OpenCV/lib/libopencv_core.so.4.1 (0x0000ffff6b6eb000) libnvinfer.so.8 => /home/×××/debug/samples/humanuid/../../lib_static/tensorrt/lib/libnvinfer.so.8 (0x0000ffff4ef13000) libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000ffff4eec7000) libcudart.so.11.0 => /usr/local/cuda-11.4/targets/aarch64-linux/include/../lib/libcudart.so.11.0 (0x0000ffff4ee12000) libstdc++.so.6 => /lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000ffff4ec2d000) libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000ffff4eb83000) libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000ffff4eb5f000) libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffff4e9ec000) /lib/ld-linux-aarch64.so.1 (0x0000ffff94a2c000) libopencv_imgproc.so.4.1 => not found libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000ffff4e9d8000) librt.so.1 => /lib/aarch64-linux-gnu/librt.so.1 (0x0000ffff4e9c0000) libnvdla_compiler.so => /usr/lib/aarch64-linux-gnu/tegra/libnvdla_compiler.so (0x0000ffff4e3fc000) libEGL.so.1 => /lib/aarch64-linux-gnu/libEGL.so.1 (0x0000ffff4e3d8000) libnvmedia.so => /usr/lib/aarch64-linux-gnu/tegra/libnvmedia.so (0x0000ffff4e370000) libnvmedia_tensor.so => /usr/lib/aarch64-linux-gnu/tegra/libnvmedia_tensor.so (0x0000ffff4e34f000) libnvmedia_dla.so => /usr/lib/aarch64-linux-gnu/tegra/libnvmedia_dla.so (0x0000ffff4e336000) libnvos.so => /usr/lib/aarch64-linux-gnu/tegra/libnvos.so (0x0000ffff4e316000) libGLdispatch.so.0 => /lib/aarch64-linux-gnu/libGLdispatch.so.0 (0x0000ffff4e18b000) libnvvideo.so => /usr/lib/aarch64-linux-gnu/tegra/libnvvideo.so (0x0000ffff4e0b4000) libnvsocsys.so => /usr/lib/aarch64-linux-gnu/tegra/libnvsocsys.so (0x0000ffff4e0a0000) libnvrm_mem.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_mem.so (0x0000ffff4e089000) libnvrm_host1x.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_host1x.so (0x0000ffff4e069000) libnvrm_surface.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_surface.so (0x0000ffff4e02b000) libnvtvmr.so => /usr/lib/aarch64-linux-gnu/tegra/libnvtvmr.so (0x0000ffff4df35000) libnvrm_chip.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_chip.so (0x0000ffff4df21000) libnvrm_sync.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_sync.so (0x0000ffff4df0b000) libnvdc.so => /usr/lib/aarch64-linux-gnu/tegra/libnvdc.so (0x0000ffff4deec000) libnvparser.so => /usr/lib/aarch64-linux-gnu/tegra/libnvparser.so (0x0000ffff4dea3000) libnvdla_runtime.so => /usr/lib/aarch64-linux-gnu/tegra/libnvdla_runtime.so (0x0000ffff4de1c000) libnvrm_stream.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_stream.so (0x0000ffff4de02000) libnvsciipc.so => /usr/lib/aarch64-linux-gnu/tegra/libnvsciipc.so (0x0000ffff4dde0000) libnvimp.so => /usr/lib/aarch64-linux-gnu/tegra/libnvimp.so (0x0000ffff4ddcb000)
在泄漏调用栈中的地址0x4AD9433B设置断点,GDB显示该地址属于动态加载后卸载的共享库/usr/local/cuda/targets/aarch64-linux/lib/libnvrtc.so:
(gdb) break *0x4AD9433B Breakpoint 1 at 0x4ad9433b (gdb) c Continuing. warning: Temporarily disabling breakpoints for unloaded shared library "/usr/local/cuda/targets/aarch64-linux/lib/libnvrtc.so" [Inferior 1 (Remote target) exited normally]
原因分析
该泄漏源于标准C++库的紧急内存池:
- 当库被静态链接时,紧急内存池会在程序退出时由系统释放,valgrind会将其标记为“仍可达”;
- 当库通过
dlopen动态加载后再用dlclose卸载时,该内存池会变为不可访问,被valgrind判定为明确泄漏。
解决方案
静态链接包含该紧急内存池的库(本场景中为libnvrtc.so),此时紧急内存池始终处于可达状态,不会被valgrind判定为明确泄漏。
内容的提问来源于stack exchange,提问作者Melina
相关产品推荐
相关产品推荐

