Azure Cognitive Search添加嵌入报错及元数据扩展问题求助

错误原因 #

报错提示属性'content'不存在于类型'search.documentFields'，核心原因是LangChain的AzureSearch默认会把Document对象的page_content映射到索引的content字段，但你的Azure搜索索引中未定义该字段。同时原代码还存在文本重复累加、函数传参错误等逻辑问题。

解决方案步骤 #

1. 对齐索引字段与LangChain映射 #

初始化AzureSearch时，需明确指定索引中存储文本内容的字段名和向量字段名，替换成你索引的实际字段：

acs = AzureSearch(
    azure_search_endpoint=AZURE_COGNITIVE_SEARCH_SERVICE_NAME,
    azure_search_key=AZURE_COGNITIVE_SEARCH_API_KEY,
    index_name=AZURE_COGNITIVE_SEARCH_INDEX_NAME,
    embedding_function=embeddings.embed_query,
    text_key="text",  # 替换为你索引中实际存储文本的字段名
    embedding_key="embedding"  # 替换为你索引中实际的向量字段名
)

2. 修正文本处理逻辑 #

原代码存在跨文件文本累加、函数传参缺失的问题，调整为单文件独立处理：

• 每个PDF单独提取完整文本，避免跨文件内容混乱
• 修复分块函数的调用参数
• 确保每个文本块正确关联对应文件的元数据

3. 添加filename元数据 #

创建LangChain的Document对象时，直接在metadata字典中加入filename字段，上传时会自动同步到Azure搜索的元数据字段中。

修正后的完整代码 #

!pip install cohere tiktoken
!pip install openai==0.28.1
!pip install pymupdf
!pip install azure-storage-blob azure-identity
!pip install azure-search-documents --pre --upgrade
!pip install langchain

import fitz
import os
import openai

from langchain.embeddings import OpenAIEmbeddings
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.chat_models import AzureChatOpenAI
from langchain.vectorstores import AzureSearch
from langchain.docstore.document import Document
from google.colab import drive

# 配置参数
OPENAI_API_BASE = "https://xxx.openai.azure.com"
OPENAI_API_KEY = "xxx"
OPENAI_API_VERSION = "2023-05-15"

openai.api_type = "azure"
openai.api_key = OPENAI_API_KEY
openai.api_base = OPENAI_API_BASE
openai.api_version = OPENAI_API_VERSION

AZURE_COGNITIVE_SEARCH_SERVICE_NAME = "https://xxx.search.windows.net"
AZURE_COGNITIVE_SEARCH_API_KEY = "xxx"
AZURE_COGNITIVE_SEARCH_INDEX_NAME = "test"

# 初始化模型和嵌入
llm = AzureChatOpenAI(
    deployment_name="gpt35",
    openai_api_key=OPENAI_API_KEY,
    openai_api_base=OPENAI_API_BASE,
    openai_api_version=OPENAI_API_VERSION
)
embeddings = OpenAIEmbeddings(
    deployment_id="ada002",
    chunk_size=1,
    openai_api_key=OPENAI_API_KEY,
    openai_api_base=OPENAI_API_BASE,
    openai_api_version=OPENAI_API_VERSION
)

# 初始化AzureSearch，指定字段映射
acs = AzureSearch(
    azure_search_endpoint=AZURE_COGNITIVE_SEARCH_SERVICE_NAME,
    azure_search_key=AZURE_COGNITIVE_SEARCH_API_KEY,
    index_name=AZURE_COGNITIVE_SEARCH_INDEX_NAME,
    embedding_function=embeddings.embed_query,
    text_key="text",  # 替换为你索引中实际的文本字段名
    embedding_key="embedding"  # 替换为你索引中实际的向量字段名
)

def split_text_into_docs(full_text, filename):
    """将完整文本分块，并创建带元数据的Document对象"""
    text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)
    splits = text_splitter.split_text(full_text)
    
    documents = []
    for idx, split in enumerate(splits):
        # 添加filename和分块索引元数据
        metadata = {
            "filename": filename,
            "chunk_index": idx
        }
        doc = Document(page_content=split, metadata=metadata)
        documents.append(doc)
    return documents

# 挂载Google Drive并处理PDF文件
drive.mount('/content/drive')
folder = "/content/drive/.../pdf/"  # 替换为你的PDF文件夹路径

for filename in os.listdir(folder):
    file_path = os.path.join(folder, filename)
    if not os.path.isfile(file_path) or not filename.endswith(".pdf"):
        continue
    
    print(f"Processing file: {filename}")
    # 提取PDF全部文本
    doc = fitz.open(file_path)
    full_text = ""
    for page in doc:
        full_text += page.get_text()
    
    # 分块并创建文档
    docs = split_text_into_docs(full_text, filename)
    # 上传到Azure Cognitive Search
    acs.add_documents(documents=docs)
    print(f"Successfully uploaded {len(docs)} chunks for {filename}")

后续查询示例 #

上传完成后，可通过以下代码实现基于ChatGPT的检索问答：

from langchain.chains import ConversationalRetrievalChain

qa_chain = ConversationalRetrievalChain.from_llm(
    llm=llm,
    retriever=acs.as_retriever(),
    return_source_documents=True
)

# 示例查询
query = "请介绍文档中的核心内容"
result = qa_chain({"question": query, "chat_history": []})
print(result['answer'])
# 查看来源文件
for doc in result['source_documents']:
    print(f"来源文件: {doc.metadata['filename']}")

内容的提问来源于stack exchange，提问作者STORM