导入Ray后出现内存泄漏求助:原因排查与报错消除方法
Ray导入后内存泄漏排查问题
在使用Ray进行并行计算时遇到内存泄漏问题,将Python脚本简化为仅包含import ray和print("just a blank script"),仍出现内存泄漏报错(以下为部分日志,其余报错均与最后一条_PyEval_EvalFrameDefault相关):
==8279== 54 bytes in 1 blocks are definitely lost in loss record 1,697 of 10,519 ==8279== at 0x4C30EDB: malloc (vg_replace_malloc.c:309) ==8279== by 0x4F23592: PyObject_Malloc (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x4F5EAE9: PyUnicode_New (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x4F675F0: _PyUnicodeWriter_PrepareInternal (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x4F3806E: PyUnicode_DecodeUTF8Stateful (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x4F7A9F7: ??? (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x4F7AFFC: ??? (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x4F876AD: _Py_BuildValue_SizeT (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x40BA46D: PyInit_setproctitle (setproctitle.c:174) ==8279== by 0x5053281: _PyImport_LoadDynamicModuleWithSpec (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x5053584: ??? (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x4FAF706: PyCFunction_Call (in /usr/lib64/libpython3.6m.so.1.0) ==8279== ==8279== 56 bytes in 1 blocks are definitely lost in loss record 3,102 of 10,519 ==8279== at 0x4C30EDB: malloc (vg_replace_malloc.c:309) ==8279== by 0x4F23592: PyObject_Malloc (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x4F3BB5C: _PyObject_GC_Malloc (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x4F4242D: _PyObject_GC_NewVar (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x4F5645E: PyTuple_New (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x4FD85E8: ??? (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x4FD8B18: ??? (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x4FD872E: ??? (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x4FD8B18: ??? (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x4FD872E: ??? (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x4FD965C: ??? (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x4FDCAA0: ??? (in /usr/lib64/libpython3.6m.so.1.0) ==8279== ==8279== 64 bytes in 1 blocks are definitely lost in loss record 3,891 of 10,519 ==8279== at 0x4C30EDB: malloc (vg_replace_malloc.c:309) ==8279== by 0x40141AD: dl_open_worker (in /usr/lib64/ld-2.28.so) ==8279== by 0x5E79AB6: _dl_catch_exception (in /usr/lib64/libc-2.28.so) ==8279== by 0x401365D: _dl_open (in /usr/lib64/ld-2.28.so) ==8279== by 0x55B81B9: dlopen_doit (in /usr/lib64/libdl-2.28.so) ==8279== by 0x5E79AB6: _dl_catch_exception (in /usr/lib64/libc-2.28.so) ==8279== by 0x5E79B52: _dl_catch_error (in /usr/lib64/libc-2.28.so) ==8279== by 0x55B8938: _dlerror_run (in /usr/lib64/libdl-2.28.so) ==8279== by 0x55B8259: dlopen@@GLIBC_2.2.5 (in /usr/lib64/libdl-2.28.so) ==8279== by 0x502DE19: _PyImport_FindSharedFuncptr (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x5053239: _PyImport_LoadDynamicModuleWithSpec (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x5053584: ??? (in /usr/lib64/libpython3.6m.so.1.0) ==8279== ==8279== 144 bytes in 1 blocks are definitely lost in loss record 7,610 of 10,519 ==8279== at 0x4C30EDB: malloc (vg_replace_malloc.c:309) ==8279== by 0x4F23592: PyObject_Malloc (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x4FCF8B2: PyCode_New (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x89FE769: __pyx_pymod_exec__raylet(_object*) (in /root/.local/lib/python3.6/site-packages/ray/_raylet.so) ==8279== by 0x502C2F2: PyModule_ExecDef (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x502C381: ??? (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x4FAF706: PyCFunction_Call (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x4FF3C41: _PyEval_EvalFrameDefault (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x4F3FBE7: ??? (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x4F795DF: ??? (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x4FBDA31: ??? (in /usr/lib64/libpython3.6m.so.1.0) ==8279== by 0x4FEDA69: _PyEval_EvalFrameDefault (in /usr/lib64/libpython3.6m.so.1.0)
使用以下命令进行内存测试:
PYTHONMALLOC=malloc valgrind --leak-check=full -v --track-origins=yes \ --log-file=$SCRIPT_DIR"/algrind_pytest.log" \ --show-possibly-lost=no \ python test.py
环境信息:
- Python 3.6.8
- Valgrind 3.15.0
- Ray 1.12.1 / 2.3.1
通过heapy检测内存使用情况:
- 未导入Ray:4721167字节
- 导入Ray后:41986865字节
- 导入Ray后执行
del ray和gc.collect():仍为41984385字节
可见导入Ray后内存占用大幅增加,且删除模块无法释放内存。
诉求:
- 了解该内存泄漏的原因
- 若并非实际问题,如何消除Valgrind的报错信息
内容的提问来源于stack exchange,提问作者faked human mao
相关产品推荐
相关产品推荐

