基于深度学习书籍的LLM聊天机器人相似性搜索返回无空格文本问题

我开发了一款基于深度学习书籍内容的LLM问答聊天机器人，已完成部署运行，但执行相似性搜索时，返回结果的所有空格都丢失了。

返回结果示例 #

nizedasacrucialtechnologythoughthefirstexperimentswithartificialneuralnetworkswereconductedinthe1950s.Deeplearninghasbeensuccessfullyusedincommercialapplicationssincethe1990s,butwasoftenregardedasbeingmoreofanartthanatechnologyandsomethingthatonlyanexpertcoulduse,untilrecently.Itistruethatsomeskillisrequiredtogetgoodperformancefromadeeplearningalgorithm.Fortunately,theamountofskillrequiredreducesastheamountoftrainingdataincreases.Thelearningalgorithmsreachinghumanperformanceoncomplextaskstodayarenearlyidenticaltothelearningalgorithmsthatstruggledtosolvetoyproblemsinthe1980s,thoughthemodelswetrainwiththesealgorithmshaveundergonechangesthatsimplifythetrainingofverydeeparchitectures.Themostimportantnewdevelopmentisthattodaywecanprovidethesealgorithmswiththeresourcestheyneedtosucceed.Figureshowshowthesizeofbenchmark1.8datasetshasincreasedremarkablyovertime.Thistrendisdrivenbytheincreasingdigitizationofsociety.Asmoreandmoreofouractivitiestakeplaceoncomputers,moreandmoreofwhatwedoisrecorded.Asourcomputersareincreasinglynetworkedtogether,itbecomeseasiertocentralizetheserecordsandcuratethem19

当前技术栈 #

• 文档拆分：TokenTextSplitter（已尝试RecursiveCharacterTextSplitter）
• 嵌入模型：text-embedding-3-large
• 向量存储：Chroma

可复现代码 #

import os
from openai import OpenAI
import chromadb
from langchain_community.document_loaders import PyPDFLoader
from langchain.text_splitter import TokenTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
import uuid

OPENAI_API_KEY = os.getenv('OPENAI_API_KEY')
OPENAI_ORGANISATION = os.getenv('OPENAI_ORGANISATION')
MODEL = os.getenv('MODEL')
CHROMA_HOST = os.getenv('CHROMA_HOST')

def connect_openai_api(organisation: str):
    client = OpenAI(
        organization=organisation
    )
    return client

def get_or_create_db(host: str, port: int):
    chroma_client = chromadb.HttpClient(host=host, port=port)
    collection = chroma_client.get_or_create_collection(name="deep_learning_chatbot")
    return collection, chroma_client

def pdf_loader(loaders: list):
    docs = []
    for loader in loaders:
        docs.extend(loader.load())
    return docs

def document_splitter(docs: list):
    text_splitter = TokenTextSplitter()
    splits = text_splitter.split_documents(docs)
    return splits

def embeddings_loader(splits: list, directory: str):
    embedding = OpenAIEmbeddings(model="text-embedding-3-large")
    chroma_client = chromadb.Client()
    vectordb =  Chroma.from_documents(
        documents=splits,
        embedding=embedding,
        persist_directory=directory
    )
    return vectordb

def run():
    client = connect_openai_api(organisation=OPENAI_ORGANISATION, model=MODEL)
    chroma_collection, client = get_or_create_db(host='localhost', port=8000)

    loaders = [
        PyPDFLoader("../docs/Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville (z-lib.org).pdf"),
    ]

    docs = pdf_loader(loaders=loaders)
    splits = document_splitter(docs=docs)
    persist_directory = "vectorDB"
    vectordb = embeddings_loader(splits=splits, directory=persist_directory)

    question = "What is deep learning?"

    docs = vectordb.similarity_search(question,k=3)

    print(docs[0].page_content)
    print(docs[1].page_content)
    print(docs[2].page_content)

解决方案 #

1. 更换PDF加载器（优先推荐） #

问题根源大概率是PyPDFLoader对目标PDF的文本提取不完整，部分PDF使用字距调整替代空格，导致提取出的文本无空格。换用PyMuPDFLoader（基于fitz库）能更准确提取带空格的文本：

• 先安装依赖：

pip install pymupdf

• 修改代码中的加载器部分：

# 替换导入
from langchain_community.document_loaders import PyMuPDFLoader

# 修改loaders列表
loaders = [
    PyMuPDFLoader("../docs/Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville (z-lib.org).pdf"),
]

2. 文本后处理补全空格 #

如果坚持使用PyPDFLoader，可以在加载文档后添加正则处理，自动补全缺失的空格：

import re

def fix_missing_spaces(text):
    # 小写字母后接大写字母时插入空格
    text = re.sub(r'([a-z])([A-Z])', r'\1 \2', text)
    # 单词与数字间插入空格
    text = re.sub(r'([a-zA-Z])(\d)', r'\1 \2', text)
    text = re.sub(r'(\d)([a-zA-Z])', r'\1 \2', text)
    # 标点后接单词时插入空格（排除句号后直接换行的情况）
    text = re.sub(r'([.,!?])([a-zA-Z])', r'\1 \2', text)
    return text

# 修改pdf_loader函数
def pdf_loader(loaders: list):
    docs = []
    for loader in loaders:
        loaded_docs = loader.load()
        for doc in loaded_docs:
            doc.page_content = fix_missing_spaces(doc.page_content)
        docs.extend(loaded_docs)
    return docs

3. 修复代码中的向量存储不一致问题 #

注意到代码中embeddings_loader使用本地Chromadb客户端，而run函数又连接了远程Chroma服务，这会导致向量存储不统一，建议统一使用远程客户端：

def embeddings_loader(splits: list, chroma_client):
    embedding = OpenAIEmbeddings(model="text-embedding-3-large")
    vectordb = Chroma.from_documents(
        documents=splits,
        embedding=embedding,
        client=chroma_client,
        collection_name="deep_learning_chatbot"
    )
    return vectordb

# 修改run函数中的调用
def run():
    # ... 其他代码 ...
    chroma_collection, chroma_client = get_or_create_db(host='localhost', port=8000)
    # ... 其他代码 ...
    vectordb = embeddings_loader(splits=splits, chroma_client=chroma_client)

内容的提问来源于stack exchange，提问作者Abdullah Bilal