如何解决LangChain ConversationalRetrievalQAChain返回无关文档来源问题？

原实现代码 #

import { OpenAI } from 'langchain/llms/openai'
import { ConversationalRetrievalQAChain } from 'langchain/chains'
import { Chroma } from 'langchain/vectorstores/chroma'
import { BufferMemory } from 'langchain/memory'

const makeChain = (vectorstore: Chroma) => {
  const model = new OpenAI({
    temperature: 0, // increase temepreature to get more creative answers
    modelName: 'gpt-3.5-turbo', // change this to gpt-4 if you have access
  })

  const questionModel = new OpenAI({})

  const chain = ConversationalRetrievalQAChain.fromLLM(
    model,
    vectorstore.asRetriever(),
    {
      returnSourceDocuments: true,
      memory: new BufferMemory({
        humanPrefix:
          'I want you to act as a document that I am having a conversation with. You will provide me with answers from the given info. If the answer is not included, search for an answer and return it. Never break character.',
        memoryKey: 'chat_history',
        inputKey: 'question', // The key for the input to the chain
        outputKey: 'text', // The key for the final conversational output of the chain
        returnMessages: true, // If using with a chat model
      }),
      questionGeneratorChainOptions: {
        llm: questionModel,
      },
    },
  )
  return chain
}

export default makeChain

问题说明 #

已实现上述ConversationalRetrievalQAChain并设置returnSourceDocuments: true，当询问文档相关问题时运行正常，但遇到通用问题（如"Who is buddha?"）或问候语（如"hi""hello"）时，系统会返回不匹配的无关文档来源，向量存储始终返回文档而不考虑查询匹配度。

解决方案 #

1. 给检索器设置相似度阈值 #

Chroma向量库支持通过检索器的searchKwargs设置相似度阈值，只有匹配度超过阈值的文档才会被返回，无匹配时返回空数组。修改vectorstore.asRetriever()部分：

vectorstore.asRetriever({
  searchKwargs: {
    k: 3, // 返回的最大文档数
    similarityThreshold: 0.7 // 相似度阈值，可根据实际情况调整
  }
})

当没有符合阈值的文档时，sourceDocuments会是空数组，此时LLM不会基于无关文档生成答案。

2. 修正Memory中的提示词逻辑 #

原humanPrefix中的"If the answer is not included, search for an answer and return it"会强制LLM寻找答案，哪怕文档中没有相关内容。修改为：

humanPrefix:
  'I want you to act as a document that I am having a conversation with. You will provide me with answers only from the given document info. If the answer is not included in the provided documents, clearly state that the information is not available in the provided documents, and you can answer general questions independently but note that the answer is not from the documents. Never break character.',

明确告知LLM在无相关文档时的处理逻辑，避免硬凑无关来源。

3. 前置问题分类处理 #

在调用对话链之前，先判断用户问题类型：

• 问候语直接返回预设回复
• 通用问题（无需文档支持）直接调用基础LLM回答，不触发检索器
• 仅文档相关问题才调用ConversationalRetrievalQAChain

示例代码片段：

const classifyQuestion = async (question: string) => {
  const classifier = new OpenAI({ temperature: 0 })
  const prompt = `Classify the following question into one of three categories: GREETING, GENERAL, DOCUMENT_RELATED.
Question: ${question}
Answer only with the category name.`
  return await classifier.call(prompt)
}

// 使用时
const category = await classifyQuestion(userQuestion)
if (category === 'GREETING') {
  return 'Hello! How can I assist you with your documents today?'
} else if (category === 'GENERAL') {
  const generalModel = new OpenAI({ temperature: 0 })
  return await generalModel.call(userQuestion)
} else {
  // 调用原对话链
  const chain = makeChain(vectorstore)
  return await chain.call({ question: userQuestion })
}

4. 使用自定义检索过滤器 #

可以自定义检索逻辑，在返回文档前过滤掉匹配度过低的结果：

const customRetriever = vectorstore.asRetriever()
customRetriever.getRelevantDocuments = async (query) => {
  const docs = await vectorstore.similaritySearchWithScore(query, 3)
  // 过滤掉相似度低于0.7的文档
  const filteredDocs = docs.filter(([doc, score]) => score >= 0.7).map(([doc]) => doc)
  return filteredDocs
}

然后在对话链中使用这个自定义检索器。

内容的提问来源于stack exchange，提问作者Basanta Rai