使用LangChain向Pinecone导入数据时线程堆积引发内存泄漏问题

数据老陈

2026-6-13

LangChain + Pinecone 嵌入数据时的线程泄漏问题

调用LangChain的add_documents方法向Pinecone嵌入数据时，每次操作会生成7个新线程（符合预期），但操作完成后这些线程始终保持活跃。后续每处理一次请求都会新增一批线程且不会被回收，长期积累后线程数量激增、内存占用持续上升，最终导致生产环境服务崩溃。

已完成的排查动作：

确认Pinecone向量存储和嵌入模型实例均以单例模式实现，不会随请求重复实例化
尝试用ThreadPoolExecutor包裹相关函数，问题仍未解决

相关代码：

pages = text_splitter.create_documents([text_input])
print(f"Active threads 2 : {[t.name for t in threading.enumerate()]}")  
combined_training_text = ""

for doc in pages:
    combined_training_text += doc.page_content + " "  # Add space between texts if needed

characters_count = count_characters(combined_training_text)
print(f"Active threads 3 : {[t.name for t in threading.enumerate()]}")  
pinecone_index = select_pinecone_manager(llm)
print(f"Active threads 4 : {[t.name for t in threading.enumerate()]}")  
vectorstore = pinecone_index.get_vectorstore(namespace_id)
print(f"Active threads 5 : {[t.name for t in threading.enumerate()]}")  
vector_ids = vectorstore.add_documents(documents=pages)
print(f"Active threads 6 : {[t.name for t in threading.enumerate()]}")  
return vector_ids, characters_count, combined_training_text[:1000]

线程日志：

Logs: Active threads 4 = ['MainThread', 'Thread-1 (start mq)',
'ThreadPoolExecutor-0_0', 'Thread-3 (worker)', 'Thread-4 (worker)',
'Thread-5 (worker)', 'Thread-6 (worker)', 'Thread-7 (handle_workers)',
'Thread-8 (handle_tasks)', 'Thread-9 (handle_results)'] Active threads
5 = ['MainThread', 'Thread-1 (start mq)', 'ThreadPoolExecutor-0_0',
'Thread-3 (worker)', 'Thread-4 (worker)', 'Thread-5 (worker)',
'Thread-6 (worker)', 'Thread-7 (handle_workers)', 'Thread-8
(handle_tasks)', 'Thread-9 (handle_results)'] Active threads 6 =
['MainThread', 'Thread-1 (start mq)', 'ThreadPoolExecutor-0_0',
'Thread-3 (worker)', 'Thread-4 (worker)', 'Thread-5 (worker)',
'Thread-6 (worker)', 'Thread-7 (handle_workers)', 'Thread-8
(handle_tasks)', 'Thread-9 (handle_results)', 'Thread-10 (worker)',
'Thread-11 (worker)', 'Thread-12 (worker)', 'Thread-13 (worker)',
'Thread-14 (handle_workers)', 'Thread-15 (handle_tasks)', 'Thread-16
(handle_results)']

有没有人遇到过类似问题？求线程清理、防止泄漏的可行建议？

内容的提问来源于stack exchange，提问作者Mohamed Hachemi