Langchain Self-querying检索仅返回4条结果，需获取全部关联文章

问题背景 #

需要从Pinecone向量库中获取所有关联sustainability标签的文章，但使用Langchain的Self-querying Retriever时，默认仅返回4条结果。已知向量库中共有7篇关联该标签的文章，手动设置limit=7可以返回全部，但由于无法提前预知匹配文章的数量，无法固定设置limit值。

用户当前代码如下：

import pinecone
from langchain.schema import Document
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import Pinecone
import os
from langchain.llms import OpenAI
from langchain.retrievers.self_query.base import SelfQueryRetriever
from langchain.chains.query_constructor.base import AttributeInfo


pinecone_api_key = "xxxxxxxx"
pinecone_env = "xxxxxxxxxx"
pinecone.init(api_key=pinecone_api_key, environment=pinecone_env)

embeddings = OpenAIEmbeddings()

index_name="langchain-self-retriever-ppo"
index = pinecone.Index(index_name)
text_field = "text"

vectorstore = Pinecone(
    index, embeddings.embed_query, text_field
)

metadata_field_info=[
    AttributeInfo(
        name="headline",
        description="The headline of the news article", 
        type="string or list[string]", 
    ),
    AttributeInfo(
        name="date",
        description="The date, news article was published", 
        type="integer", 
    ),
    AttributeInfo(
        name="publication",
        description="The name of the publication which published this news article", 
        type="string", 
    ),
    AttributeInfo(
        name="domain",
        description="The domain of the news article",
        type="float"
    ),
]
document_content_description = "Brief summary of a news article"
llm = OpenAI(temperature=0)

retriever = SelfQueryRetriever.from_llm(
    llm, 
    vectorstore, 
    document_content_description, 
    metadata_field_info, 
    enable_limit=True,
    verbose=True
)

# This example only specifies a relevant query
retrieved_docs = retriever.get_relevant_documents("Articles which are related to sustainability")
print(retrieved_docs)
print(len(retrieved_docs))

排查发现：get_relevant_documents方法调用self.vectorstore.search，进而调用self.similarity_search，该方法默认将limit设为4。

解决方案 #

方法1：动态获取匹配文档总数再设置limit #

先通过Pinecone原生查询接口统计匹配指定标签的文档数量，再将该数量作为查询的limit值传入。

# 构造过滤条件（根据实际metadata字段调整，假设标签存储在headline字段）
filter_query = {"headline": {"$in": ["sustainability"]}}

# 执行Pinecone原生查询，获取匹配文档总数
query_response = index.query(
    vector=embeddings.embed_query("sustainability"),  # 用任意向量占位，仅需统计数量
    filter=filter_query,
    top_k=1,
    include_metadata=True,
    include_values=False
)
total_matches = query_response['total']

# 传入动态获取的总数作为limit，查询所有匹配文档
retrieved_docs = retriever.get_relevant_documents(
    "Articles which are related to sustainability",
    search_kwargs={"k": total_matches}
)

print(retrieved_docs)
print(len(retrieved_docs))

方法2：自定义Retriever子类修改默认limit #

继承SelfQueryRetriever类，重写相关方法，将默认查询数量设为向量库总文档数，确保能返回所有匹配结果。

from langchain.retrievers.self_query.base import SelfQueryRetriever

class UnlimitedSelfQueryRetriever(SelfQueryRetriever):
    def _get_relevant_documents(self, query):
        # 获取向量库总文档数，作为查询的默认limit
        total_docs = index.describe_index_stats()['total_vector_count']
        self.search_kwargs["k"] = total_docs
        return super()._get_relevant_documents(query)

# 初始化自定义Retriever
retriever = UnlimitedSelfQueryRetriever.from_llm(
    llm, 
    vectorstore, 
    document_content_description, 
    metadata_field_info, 
    enable_limit=True,
    verbose=True
)

retrieved_docs = retriever.get_relevant_documents("Articles which are related to sustainability")
print(retrieved_docs)
print(len(retrieved_docs))

方法3：直接使用向量库的过滤查询方法 #

绕过SelfQueryRetriever，直接调用Pinecone向量库的similarity_search_with_filter方法，传入过滤条件并设置足够大的查询数量。

# 构造过滤条件（根据实际metadata字段调整）
filter_query = {"headline": {"$in": ["sustainability"]}}
# 获取向量库总文档数作为查询上限
total_docs = index.describe_index_stats()['total_vector_count']

# 直接执行过滤查询
retrieved_docs = vectorstore.similarity_search_with_filter(
    query="sustainability",
    filter=filter_query,
    k=total_docs
)

print(retrieved_docs)
print(len(retrieved_docs))

内容的提问来源于stack exchange，提问作者Shubham Shinde