Langchain检索器过滤问题：BM25在EnsembleRetriever中如何维持过滤规则

LangChain中部分检索器（比如FAISS向量检索器）支持filter参数，但BM25Retriever默认不支持原生过滤逻辑，所以你当前的代码里只有FAISS检索器应用了topic="sport"的过滤，BM25检索器仍会返回所有文档，导致组合后的EnsembleRetriever整体过滤规则失效。

要让BM25检索器也维持过滤规则，有两种可行方案：

方案1：预过滤文档后初始化BM25Retriever #

先从原始文档集中筛选出符合过滤条件的文档，再用这些文档构建BM25Retriever，这样BM25检索的范围本身就限定在目标文档内：

documents = [Document(page_content='The Celtics are my favourite team.', metadata={"topic":"sport"}),
     Document(page_content='The Boston Celtics won the game by 20 points', metadata={"topic":"sport"}),
     Document(page_content='This is just a random text.', metadata={"topic":"unknown"})]

# 预过滤出topic为sport的文档
filtered_docs = [doc for doc in documents if doc.metadata.get("topic") == "sport"]

# embeddings 为任意LangChain嵌入模型
db = FAISS.from_documents(documents, embeddings) 
question = "Who is my favourite team?"

# 用过滤后的文档初始化BM25Retriever
retriever = BM25Retriever.from_documents(filtered_docs)
faiss_retriever = db.as_retriever(search_kwargs={'filter': dict(topic="sport"), 'k': 4, 'fetch_k': 8})
er = EnsembleRetriever(retrievers=[retriever, faiss_retriever], weights=[0.3, 0.7])
results = er.get_relevant_documents(question)

方案2：自定义带过滤功能的BM25Retriever子类 #

继承原BM25Retriever类，重写检索方法，在检索前先对文档应用过滤规则：

from langchain.retrievers import BM25Retriever

class FilterableBM25Retriever(BM25Retriever):
    def __init__(self, *args, filter_dict=None, **kwargs):
        super().__init__(*args, **kwargs)
        self.filter_dict = filter_dict or {}

    def get_relevant_documents(self, query, **kwargs):
        # 先过滤文档
        filtered_docs = []
        for doc in self.documents:
            match = True
            for key, value in self.filter_dict.items():
                if doc.metadata.get(key) != value:
                    match = False
                    break
            if match:
                filtered_docs.append(doc)
        # 临时替换原文档为过滤后的文档
        original_docs = self.documents
        self.documents = filtered_docs
        # 执行BM25检索
        results = super().get_relevant_documents(query, **kwargs)
        # 恢复原文档
        self.documents = original_docs
        return results

# 使用自定义检索器
documents = [Document(page_content='The Celtics are my favourite team.', metadata={"topic":"sport"}),
     Document(page_content='The Boston Celtics won the game by 20 points', metadata={"topic":"sport"}),
     Document(page_content='This is just a random text.', metadata={"topic":"unknown"})]

# embeddings 为任意LangChain嵌入模型
db = FAISS.from_documents(documents, embeddings) 
question = "Who is my favourite team?"

# 初始化带过滤的BM25Retriever
retriever = FilterableBM25Retriever.from_documents(documents, filter_dict={"topic":"sport"})
faiss_retriever = db.as_retriever(search_kwargs={'filter': dict(topic="sport"), 'k': 4, 'fetch_k': 8})
er = EnsembleRetriever(retrievers=[retriever, faiss_retriever], weights=[0.3, 0.7])
results = er.get_relevant_documents(question)

两种方案都能确保BM25检索器只返回符合过滤条件的文档，和FAISS检索器的结果统一后，EnsembleRetriever就能维持整体的过滤规则。

内容的提问来源于stack exchange，提问作者tcotts