Discord Bot集成pdfplumber出现内存占用过高问题求助

核心问题定位 #

你遇到的内存泄漏，本质是pdfplumber处理页面后部分对象仍被引用无法被GC回收，即便手动调用gc.collect()也无法彻底清理，尤其是异步环境下用run_in_executor执行同步代码时，对象引用可能跨线程残留。

针对性修复方案 #

1. 用子进程隔离PDF处理逻辑 #

线程共享内存空间易导致对象引用残留，改用子进程执行PDF处理，子进程结束后会释放所有内存，从根源避免泄漏：

import multiprocessing

async def ext_command(self, ctx:interactions.CommandContext, page: int = None):
    await ctx.defer(ephemeral=False)
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
            r = await response.content.read()
            # 使用子进程池执行PDF处理
            with multiprocessing.Pool(1) as pool:
                chunk = pool.apply(self.pdf_process, (r, page))
            for part in chunk:
                await ctx.send(part)

2. 简化PDF处理中的清理逻辑 #

原代码手动清理req_page._objects()等操作不仅无效，还可能破坏对象结构，直接移除这些操作，依赖with上下文管理器自动释放资源：

def pdf_process(self, r, page):
    extracted_page = ""
    with pdfplumber.open(BytesIO(r)) as pdf:
        if page is not None:
            extracted_page = pdf.pages[page-1].extract_text() or ""
        else:
            current_day = datetime.datetime.now().timetuple().tm_yday
            current_month = datetime.date.today().month
            page = current_day + current_month + 7
            year = datetime.date.today().year
            if (year % 4 == 0 and year % 100 != 0) or year % 400 == 0:
                page -= 1
            # 直接提取文本，不保留page对象引用
            extracted_page = pdf.pages[page-1].extract_text() or ""
    
    chunklength = 1900
    chunk = ""
    label = 0
    indexes = [i for i, letter in enumerate(extracted_page) if letter == '\n']
    for idx in indexes:
        if idx >= 2 and extracted_page[idx-1] == ' ':
            prev_char = extracted_page[idx-2]
            if prev_char in ('.', '"', '!', ' ', '?'):
                if label == 0:
                    chunk = extracted_page[:idx] + '\n'
                else:
                    chunk += extracted_page[label:idx] + '\n'
                label = idx
    if label < len(extracted_page):
        chunk += extracted_page[label:-1] if len(extracted_page) > 1 else extracted_page[label:]
    
    chunks = [chunk[i:i+chunklength] for i in range(0, len(chunk), chunklength)]
    return chunks

3. 显式释放大对象引用 #

异步函数中处理完PDF字节流后，显式将其赋值为None，帮助GC快速识别可回收对象：

async def ext_command(self, ctx:interactions.CommandContext, page: int = None):
    await ctx.defer(ephemeral=False)
    r = None
    try:
        async with aiohttp.ClientSession() as session:
            async with session.get(url) as response:
                r = await response.content.read()
                with multiprocessing.Pool(1) as pool:
                    chunk = pool.apply(self.pdf_process, (r, page))
            for part in chunk:
                await ctx.send(part)
    finally:
        r = None  # 显式释放PDF字节流引用

4. 升级pdfplumber到最新版本 #

pdfplumber后续版本修复了不少内存泄漏问题，执行pip install --upgrade pdfplumber更新，可能无需额外代码即可解决问题。

验证方法 #

用memory_profiler再次分析内存变化：

1. 安装依赖：pip install memory-profiler
2. 在ext_command函数前添加@profile装饰器
3. 多次执行命令，观察内存是否不再持续上涨

内容的提问来源于stack exchange，提问作者Parth