如何让LangChain的get_relevant_documents仅在答案来自自定义知识库时返回结果

开发一个可访问外部知识库docs的聊天机器人，需求如下：

• 当用户提问需要依赖知识库内容时，返回相关文档用于生成答案
• 当用户输入问候语（如“hello”“how are you”）、简单算术（如“2+2等于多少”）这类无需调用知识库的内容时，检索器返回空列表

尝试设置检索器的相似度阈值后出现异常：无关内容仍返回高相似度文档，而需要知识库的场景却无法获取到相关文档。

1. 通过LLM判断查询是否需要调用知识库 #

新增分类逻辑，用LLM对用户查询进行判断，明确是否需要从外部知识库获取信息，再决定是否调用检索器。

def should_retrieve(query: str) -> bool:
    # 定义判断提示词，让LLM仅返回是/否
    prompt = PromptTemplate(
        input_variables=["query"],
        template="判断以下用户查询是否需要从外部知识库获取信息来回答，仅返回'是'或'否'：\n{query}"
    )
    response = llm.predict(prompt.format(query=query))
    return response.strip() == "是"

使用时先判断再调用检索器：

query = "2+2等于多少"
if should_retrieve(query):
    relevant_docs = retriever.get_relevant_documents(query)
else:
    relevant_docs = []
print(relevant_docs)  # 会返回空列表

2. 优化相似度阈值设置 #

之前设置的score_threshold=0.9过高，OpenAI Embedding的余弦相似度通常在0.5-0.8区间代表较强相关性，调整阈值并配合返回数量参数k，可以平衡召回率和准确率。

# 调整阈值为0.7，同时设置最多返回3条相关文档
retriever = vectorstore.as_retriever(
    search_type="similarity_score_threshold",
    search_kwargs={"score_threshold": 0.7, "k": 3}
)

3. 结合两种方法实现双重过滤 #

将分类判断和阈值检索结合，先判断是否需要检索，再用优化后的阈值检索器获取文档，既避免无关查询调用知识库，又保证相关查询的文档召回质量。

修改后的完整代码示例：

import os
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import FAISS
from langchain.chains import ConversationalRetrievalChain 
from langchain.memory import ConversationBufferMemory
from langchain.chat_models import ChatOpenAI
from langchain.prompts import PromptTemplate

os.environ['OPENAI_API_KEY'] = '你的API密钥'

custom_template = """
This is conversation with a human. Answer the questions you get based on the knowledge you have.
If you don't know the answer, just say that you don't, don't try to make up an answer.
Chat History:
{chat_history}
Follow Up Input: {question}
"""
CUSTOM_QUESTION_PROMPT = PromptTemplate.from_template(custom_template)

llm = ChatOpenAI(
    model_name="gpt-3.5-turbo",
    temperature=0
)

# 判断是否需要调用知识库的函数
def should_retrieve(query: str) -> bool:
    prompt = PromptTemplate(
        input_variables=["query"],
        template="判断以下用户查询是否需要从外部知识库获取信息来回答，仅返回'是'或'否'：\n{query}"
    )
    response = llm.predict(prompt.format(query=query))
    return response.strip() == "是"

embeddings = OpenAIEmbeddings()

docs = [
    "Buildings are made out of brick",
    "Buildings are made out of wood",
    "Buildings are made out of stone",
    "Buildings are made out of atoms",
    "Buildings are made out of building materials",
    "Cars are made out of metal",
    "Cars are made out of plastic",
  ]

vectorstore = FAISS.from_texts(docs, embeddings)

# 优化后的检索器
retriever = vectorstore.as_retriever(
    search_type="similarity_score_threshold",
    search_kwargs={"score_threshold": 0.7, "k": 3}
)

memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)

qa = ConversationalRetrievalChain.from_llm(
    llm,
    retriever,
    condense_question_prompt=CUSTOM_QUESTION_PROMPT,
    memory=memory
)

# 测试不同查询
test_queries = ["what are cars made of?", "hello", "2+2等于多少"]
for query in test_queries:
    print(f"\n查询: {query}")
    if should_retrieve(query):
        result = qa({"question": query})
        print("回答:", result['answer'])
        print("相关文档:", retriever.get_relevant_documents(query))
    else:
        # 直接用LLM回答无需知识库的问题
        direct_answer = llm.predict(query)
        print("回答:", direct_answer)
        print("相关文档: []")

内容的提问来源于stack exchange，提问作者Blue Cheese