BM25+PgVector混合检索精度未达预期，如何优化？

问题场景 #

我构建了一个基于关键词+上下文的文档检索原型，采用all-mpnet-base-v2生成向量嵌入并存储于PGVector，通过EnsembleRetriever结合BM25（关键词检索）与PGVector（上下文检索）实现混合检索。但针对问题**「What is meant by expect_column_values_to_be_between?」**，数据库中明明存在相关文档（内容如下），却无法召回预期结果。

数据库中的相关文档内容 #

The content contains information about 'expect_column_values_to_be_between', which is a sub-title coming under a hierarchy of titles as ['Rule library']. The actual content starts from here: Description: Validates that entries in a specified column fall within a defined inclusive range, ensuring data adheres to expected bounds.
Dimension: Accuracy
Rule Level: Column
Mandatory Argument(s):
1.Column name (Supported data types: Numeric )
2.Enter lower bound (Supported data types: Numeric )
3.Enter upper bound (Supported data types: Numeric )
Optional Argument(s):
1.Value to be greater than lower bound ( When switched ON, rule succeeds only when value is strictly greater than specified lower bound value; when OFF, rule succeeds even when the value is greater than or equal to the specified lower bound value. By default, it is switched OFF )
2.Value to be lesser than upper bound ( When switched ON, rule succeeds only when value is strictly less than specified upper bound value; when OFF, rule succeeds even when the value is lower than or equal to the specified upper bound value. By default, it is switched OFF )
3.Tolerance level (%) ( Percentage of records that is expected to meet the required criteria, below which the rule fails. By default it is set to 100%, meaning all records are expected to meet specified criteria )

原实现代码 #

def hybrid_search(question):
    embeddings = HuggingFaceEmbeddings(model_name="all-mpnet-base-v2")
    df_ = pd.read_csv("contents.csv", usecols=["enhancedContent"])
    loader_ = DataFrameLoader(df_, page_content_column='enhancedContent')
    docs = loader_.load()
    pages = loader_.load_and_split()

    bm25_retriever = BM25Retriever.from_documents(pages)
    bm25_retriever.k = 2  

    collection_name = "dummy_db" 
    CONNECTION_STRING = config.get("pg_vector_details", "CONNECTION_STRING") % quote_plus(
        config.get("pg_vector_details", "password"))

    store1 = PGVector(
        collection_name=collection_name,
        connection_string=CONNECTION_STRING,
        embedding_function=embeddings, )

    retriever_pgvector = store1.as_retriever(
        search_kwargs={"k": 3}
    )

    ensemble_retriever = EnsembleRetriever(retrievers=[bm25_retriever, 
     retriever_pgvector], weights=[0.4, 0.6])

    context = ensemble_retriever.get_relevant_documents(question)
    print("Context from DB- Ensemble retriever: ", context)
    return context, ensemble_retriever

问题分析 #

无法召回的核心原因可能包括：

• 文档拆分逻辑不当，导致目标术语所在的完整上下文被拆分为过小片段，无法被检索器匹配
• BM25召回数量过少，漏过相关结果；权重设置未发挥其关键词匹配的优势
• 向量检索的召回范围不足，或文本预处理缺失导致嵌入相似度偏低

针对性修改方案 #

1. 修复文档拆分逻辑 #

原代码中load_and_split()会默认拆分文档，可能破坏包含目标术语的完整上下文。改为直接加载完整文档，确保检索单元的完整性：

# 替换原有的docs和pages加载逻辑
pages = loader_.load()  # 不拆分，直接加载完整文档

2. 调整检索器参数 #

• 优化BM25召回数量：术语类问题对关键词匹配敏感度高，调高BM25的召回范围：

bm25_retriever.k = 5  # 从2提升至5，扩大候选池

• 扩大PGVector召回范围：确保相关向量结果被纳入候选：

retriever_pgvector = store1.as_retriever(search_kwargs={"k": 5})

3. 调整EnsembleRetriever权重 #

针对术语定义类问题，提高BM25的权重，让关键词匹配发挥主导作用：

ensemble_retriever = EnsembleRetriever(
    retrievers=[bm25_retriever, retriever_pgvector],
    weights=[0.6, 0.4]  # 交换权重，BM25占比提升至0.6
)

4. 添加文本预处理增强匹配 #

统一文档与问题的文本格式，避免大小写等因素影响嵌入和检索结果：

def preprocess_text(text):
    return text.strip().lower()

embeddings = HuggingFaceEmbeddings(
    model_name="all-mpnet-base-v2",
    encode_kwargs={"normalize_embeddings": True},
    preprocess_function=preprocess_text
)

5. 验证PGVector嵌入有效性 #

单独测试PGVector检索，确认文档是否正确存入并生成嵌入：

# 单独测试PGVector检索
test_docs = retriever_pgvector.get_relevant_documents("What is meant by expect_column_values_to_be_between?")
print("PGVector单独检索结果:", test_docs)

若单独检索也无结果，需检查文档是否正确写入PGVector，或嵌入生成过程是否存在异常。

其他可行方案 #

方案一：CrossEncoder重排序 #

在EnsembleRetriever召回候选后，用CrossEncoder对候选文档与问题做精准相似度重排序，提升最终结果的准确性：

from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import CrossEncoderReranker
from langchain.embeddings import HuggingFaceCrossEncoder

# 初始化CrossEncoder重排序器
cross_encoder = HuggingFaceCrossEncoder(model_name="cross-encoder/ms-marco-MiniLM-L-6-v2")
compressor = CrossEncoderReranker(model=cross_encoder, top_n=3)

# 构建带重排序的检索器
compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=ensemble_retriever
)

# 获取重排序后的结果
context = compression_retriever.get_relevant_documents(question)

方案二：自定义BM25分词逻辑 #

默认BM25分词可能拆分下划线连接的技术术语，自定义分词器保留术语完整性：

from rank_bm25 import BM25Okapi
from langchain.retrievers import BM25Retriever

# 自定义分词函数，按空格拆分以保留完整术语
def custom_tokenizer(text):
    return text.split()

# 手动初始化BM25Retriever
tokenized_docs = [custom_tokenizer(doc.page_content) for doc in pages]
bm25 = BM25Okapi(tokenized_docs)
bm25_retriever = BM25Retriever(
    bm25=bm25,
    docs=pages,
    tokenizer=custom_tokenizer
)
bm25_retriever.k = 5

内容的提问来源于stack exchange，提问作者Bhavya