Jupyter中urllib parse占GB级内存，排查发现根源在pandas

问题背景 #

循环每日发送GET请求拉取单天约200MB的文本数据，已启用gc并手动删除所有变量，但tracemalloc显示urllib.parse占用大量内存。先后尝试用with语句管理资源、切换至requests库，问题仍存在，最终确认根源在Pandas操作。

初始代码 #

import tracemalloc

tracemalloc.start()

import pandas as pd
import json
import gc
from datetime import datetime, timedelta,date
from urllib.request import Request, urlopen  # Python 3
gc.enable()

def daterange(start_date, end_date):
    for n in range(int((end_date - start_date).days)):
        yield start_date + timedelta(n)

start_date = date(2022, 8, 1)
end_date = date(2022, 8, 7)

def func(single_date):
    print(single_date.strftime("%Y-%m-%d"))
    date = single_date.strftime("%Y-%m-%d")
    req = Request('https://website.com/export?from_date='+date+'&to_date='+date+'&project_id=00000')
    req.add_header('Accept', 'text/plain')
    req.add_header( 'Authorization', 'Basic ==')
    content = urlopen(req)
    encoding = content.info().get_content_charset('utf-8')
    data_object = content.read()
    data = data_object.decode(encoding)
    
    df = pd.read_json(data, lines=True, orient='records')
    dfout = pd.concat([df.reset_index()[['event']], pd.json_normalize(df['properties'])], axis=1)

    del df
    del dfout
    del req
    del content
    del data_object
    del data
    del date
    del encoding
    del single_date
    

for single_date in daterange(start_date, end_date):
    gc.collect()
    func(single_date)

snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')

print("[ Top 10 ]")
for stat in top_stats[:10]:
    print(stat)

tracemalloc输出 #

[ Top 10 ]
/opt/tljh/user/lib/python3.9/urllib/parse.py:461: size=1391 MiB, count=6, average=232 MiB
/opt/tljh/user/lib/python3.9/urllib/parse.py:486: size=1253 MiB, count=6, average=209 MiB
/opt/tljh/user/lib/python3.9/urllib/parse.py:488: size=132 MiB, count=24, average=5628 KiB
<frozen importlib._bootstrap_external>:647: size=12.0 MiB, count=115719, average=108 B
<frozen importlib._bootstrap>:228: size=5055 KiB, count=24566, average=211 B
/opt/tljh/user/lib/python3.9/tracemalloc.py:505: size=754 KiB, count=13788, average=56 B
/opt/tljh/user/lib/python3.9/tracemalloc.py:498: size=647 KiB, count=13788, average=48 B
/opt/tljh/user/lib/python3.9/tracemalloc.py:193: size=646 KiB, count=13787, average=48 B
/home/jupyter/.local/lib/python3.9/site-packages/pandas/util/_decorators.py:389: size=366 KiB, count=256, average=1463 B
/opt/tljh/user/lib/python3.9/re.py:210: size=277 KiB, count=181, average=1568 B

运行6天数据后，Notebook内存占用约4GB。

已尝试的解决方案 #

• 更新1：改用with语句管理资源，内存问题仍存在：

def func(single_date):
    print(single_date.strftime("%Y-%m-%d"))
    date = single_date.strftime("%Y-%m-%d")
    req = Request('https://website.com/export?from_date='+date+'&to_date='+date+'&project_id=00000')
    req.add_header('Accept', 'text/plain')
    req.add_header( 'Authorization', 'Basic ==')
    with urlopen(req) as content:
        encoding = content.info().get_content_charset('utf-8')
        data_object = content.read()  # data is now bytes
        data = data_object.decode(encoding)
        df = pd.read_json(data, lines=True, orient='records')
        dfout = pd.concat([df.reset_index()[['event']],  
        pd.json_normalize(df['properties'])], axis=1)

• 更新2：切换至requests库，结果相同：

with requests.get(url, headers=headers) as r:
            df = pd.read_json(r.text, lines=True, orient='records')
            dfout = pd.concat([df.reset_index()[['event']], pd.json_normalize(df['properties'])], axis=1)

• 更新3：确认问题根源在Pandas

解决方向建议 #

1. 分块读取数据：使用pd.read_json的chunksize参数，将200MB数据分成小块逐批处理，避免一次性加载全量数据到内存：

   for chunk in pd.read_json(data, lines=True, orient='records', chunksize=10000):
       # 对每个chunk执行处理逻辑
       chunk_out = pd.concat([chunk[['event']], pd.json_normalize(chunk['properties'])], axis=1)
       # 处理后直接写入文件或进行后续操作，不保存全量DataFrame
       chunk_out.to_csv('output.csv', mode='a', header=False)
   

2. 优化DataFrame操作：

   • 避免不必要的中间DataFrame：比如df.reset_index()[['event']]可简化为df[['event']]（无需重置索引时）；
   • 减少DataFrame复制操作，尽量在原数据上直接处理，或使用copy=False（适用场景）；
   • 对pd.json_normalize结果仅保留需要的字段，避免加载冗余数据。

3. 强化内存释放：在函数内部处理完数据后，立即删除相关变量并强制触发垃圾回收，确保内存及时释放：

   def func(single_date):
       # 数据拉取与处理逻辑...
       df = pd.read_json(...)
       dfout = pd.concat(...)
       
       # 处理完成后立即清理
       del df, dfout
       gc.collect()
   

4. 升级Pandas版本：部分旧版本Pandas在read_json或json_normalize中存在内存泄漏问题，升级到最新稳定版可能修复该问题。

5. 改用轻量JSON解析：先用Python标准库json逐行解析原始数据，只提取需要的event和properties字段，再构建DataFrame，减少Pandas处理负载：

   import json
   data_lines = data.splitlines()
   parsed_data = []
   for line in data_lines:
       obj = json.loads(line)
       parsed_data.append({
           'event': obj['event'],
           **obj['properties']
       })
   df = pd.DataFrame(parsed_data)
   

内容的提问来源于stack exchange，提问作者ivan-orange