如何通过LangChain为VertexAI搜索查询添加元数据过滤

问题背景 #

我已经将带元数据的文档上传至VertexAI Search and Conversation，正在用LangChain构建RAG模型，希望依据上传的元数据（格式如下）过滤生成回答的文档——比如只使用language为English的文档，但给GoogleVertexAISearchRetriever添加filter参数时一直报错，求解决方法。

元数据格式 #

{"id": 1,
 "structData": {"id": "2743", 
                "tags": "Operation",
                "language": "English"
                },
 "content": {"mimeType": "text/plain", "uri": "#link to gcs location of the file"}}

现有代码 #

import vertexai
from langchain.llms import VertexAI
from langchain.chains import RetrievalQA
from langchain.retrievers import GoogleVertexAISearchRetriever

vertexai.init(project=PROJECT_ID, location=REGION)
llm = VertexAI(model_name=MODEL)

retriever = GoogleVertexAISearchRetriever(
    project_id=PROJECT_ID,
    location_id=DATA_STORE_LOCATION,
    data_store_id=DATA_STORE_ID,
    get_extractive_answers=True,
    max_documents=10,
    max_extractive_segment_count=1,
    max_extractive_answer_count=5
)

search_query = "What are some spices that can help lower blood sugar?"

retrieval_qa = RetrievalQA.from_chain_type(
    llm=llm, chain_type="stuff", retriever=retriever, return_source_documents=True
)

results = retrieval_qa({"query": search_query})

解决方案 #

核心要点 #

GoogleVertexAISearchRetriever的filter参数必须遵循VertexAI Search的结构化元数据过滤规则：针对上传时的structData字段，在filter中需要用struct_data.作为前缀来指定字段路径，同时注意字符串值的引号格式。

修正后的代码 #

import vertexai
from langchain.llms import VertexAI
from langchain.chains import RetrievalQA
from langchain.retrievers import GoogleVertexAISearchRetriever

vertexai.init(project=PROJECT_ID, location=REGION)
llm = VertexAI(model_name=MODEL)

# 添加filter参数，指定仅使用language为English的文档
retriever = GoogleVertexAISearchRetriever(
    project_id=PROJECT_ID,
    location_id=DATA_STORE_LOCATION,
    data_store_id=DATA_STORE_ID,
    get_extractive_answers=True,
    max_documents=10,
    max_extractive_segment_count=1,
    max_extractive_answer_count=5,
    # 关键：用struct_data前缀访问结构化元数据的language字段
    filter='struct_data.language = "English"'
)

search_query = "What are some spices that can help lower blood sugar?"

retrieval_qa = RetrievalQA.from_chain_type(
    llm=llm, chain_type="stuff", retriever=retriever, return_source_documents=True
)

results = retrieval_qa({"query": search_query})

额外说明 #

• 如果需要多条件过滤，可以用逻辑运算符组合，比如：

  filter='struct_data.language = "English" AND struct_data.tags = "Operation"'
  

• 数值类型的字段不需要加引号，比如struct_data.id = 2743

内容的提问来源于stack exchange，提问作者martinrame