如何用tracemalloc修复Flask应用的内存泄漏问题

问题背景 #

我开发的Flask应用通过请求获取数据，再使用openpyxl将数据导出为Excel文件。但在导出约50个Excel文件后，应用内存占用超过8GB并崩溃。

内存检测代码 #

我使用tracemalloc编写了内存检测函数，每次导出Excel后调用它：

import tracemalloc
tracemalloc.start()

def get_allocated_memory():
    """
    Prints allocated memmory at time of function call in log file.
    """

    snapshot = tracemalloc.take_snapshot()
    top_stats = snapshot.statistics('lineno')

    logger.debug("========== ALLOCATED MEMMORY =============")
    for stat in top_stats[:10]:
        logger.debug(str(stat))
        logger.debug(str(stat.traceback.format()))

首次导出50MB Excel后的检测结果 #

========== ALLOCATED MEMMORY =============
/Users/.../opt/anaconda3/lib/python3.9/json/decoder.py:353: size=108 MiB, count=994668, average=113 B
['  File "/Users/.../opt/anaconda3/lib/python3.9/json/decoder.py", line 353', '    obj, end = self.scan_once(s, idx)']
/Users/i.../PIR.py:147: size=7475 KiB, count=54234, average=141 B
['  File "/Users/.../PIR.py", line 147', '    dict_category[\'/glossary/\' + replaced] = entry[\'rep\'][\'title\'].replace("&", "&") + \'|\' + \\']
/Users/.../opt/anaconda3/lib/python3.9/ssl.py:1124: size=5120 KiB, count=1, average=5120 KiB
['  File "/Users/.../opt/anaconda3/lib/python3.9/ssl.py", line 1124', '    return self._sslobj.getpeercert(binary_form)']
/Users/.../PIR.py:143: size=4614 KiB, count=30535, average=155 B
['  File "/Users/...PIR.py", line 143', '    dict_category[\'/glossary/\' + entry[\'rep\'][\'id\']] = entry[\'rep\'][\'title\'].replace("&", "&") + \'|\' + \\']
/Users/.../PIR.py:155: size=3146 KiB, count=89489, average=36 B
['  File "/Users/.../PIR.py", line 155', "    replaced] = [entry['rep']['title'], entry['rep']]"]
/Users/.../PIR.py:151: size=1738 KiB, count=49430, average=36 B
['  File "/Users/i.../PIR.py", line 151', "    ] = [entry['rep']['title'], entry['rep']]"]
<frozen importlib....>:647: size=1473 KiB, count=17692, average=85 B
['  File "<frozen importlib._bootstrap_external>", line 647']
/Users/..../opt/anaconda3/lib/python3.9/site-packages/openpyxl/utils/cell.py:94: size=926 KiB, count=18252, average=52 B
['  File "/Users/.../opt/anaconda3/lib/python3.9/site-packages/openpyxl/utils/cell.py", line 94', "    return ''.join(reversed(letters))"]
/Users/.../opt/anaconda3/lib/python3.9/site-packages/openpyxl/descriptors/__init__.py:13: size=796 KiB, count=2685, average=304 B
['  File "/Users/.../opt/anaconda3/lib/python3.9/site-packages/openpyxl/descriptors/__init__.py", line 13', '    return type.__new__(cls, clsname, bases, methods)']
<frozen importlib._bootstrap>:228: size=673 KiB, count=6360, average=108 B
['  File "<frozen importlib._bootstrap>", line 228']

解决建议 #

• 清理JSON解析后的临时数据：JSON解析占用了最大内存（108MiB），检查是否有全局变量持有解析后的JSON对象，每次请求处理完成后及时删除或清空这些对象，避免内存累积。
• 重置全局字典dict_category：从检测结果看，dict_category相关操作占用了大量内存，若该字典是全局变量，每次导出Excel后需清空不再需要的键值对，或改为局部变量仅在单次导出流程中使用，避免数据持续累积。
• 正确释放openpyxl资源：使用openpyxl时，通过上下文管理器（with语句）创建和操作Workbook，确保资源自动释放：

  from openpyxl import Workbook
  
  with Workbook() as wb:
      # 执行Excel写入操作
      wb.save('output.xlsx')
  

• 关闭HTTP请求连接：SSL相关内存占用可能来自未正确关闭的请求连接，使用requests.Session的上下文管理器来管理请求，确保连接自动关闭：

  import requests
  
  with requests.Session() as session:
      response = session.get(api_url)
      # 处理响应数据
  

• 避免全局容器累积：检查代码中所有全局列表、字典等容器，确认每次导出后是否需要重置，防止数据不断堆积。
• 辅助垃圾回收：在每次导出完成后，手动调用gc.collect()辅助回收内存，但这仅为补充手段，核心仍需解决内存泄漏根源。

内容的提问来源于stack exchange，提问作者David