OpenAI+FAISS集成异常：模型未调用自定义Excel数据且存在属性错误

从Azure Blob Storage加载Excel数据后，终端显示加载成功，但调用模型接口时，模型返回通用内容而非自定义数据。使用RecursiveCharacterTextSplitter修复了OpenAI和FAISS模块的'str' object has no attribute 'page_content'错误，但模型仍无法使用自定义数据；移除该分割器则会再次触发上述属性错误，怀疑组件未正确传入模型。

class MyBot(ActivityHandler):
    def __init__(self, conversation_state: ConversationState):

        # Initialize the BlobServiceClient with your storage account connection string
        try:
            
            connection_str = "from azure"
            blob_service_client = BlobServiceClient.from_connection_string(connection_str)

        # Define your blob container and blob name
            container_name = "dataset"
            blob_name = "data.xlsx"

        # Get blob client
            blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_name)

        # Download blob content as string for csv or bytes for xlsx
            #blob_content_as_str = blob_client.download_blob().readall().decode('utf-8')
            blob_content_as_bytes = blob_client.download_blob().readall()

        # Convert the string data to pandas DataFrame
            #csv_buffer = StringIO(blob_content_as_str)
            #data = pd.read_csv(csv_buffer)
            excel_buffer  = BytesIO(blob_content_as_bytes)
            data = pd.read_excel(excel_buffer,sheet_name='Sheet1',engine='openpyxl')
            print(data.head(5))
            data = data.iloc[:,-2:]
            print(data.head(5))

        # Show DataFrame
            print(type(data))
            

        except AttributeError as e:    
            print("AttributeError occurred in Blob storage part:", e)
        
        try:
        # OpenAI and FAISS related code
 
            text_splitter = RecursiveCharacterTextSplitter(chunk_size = 1,chunk_overlap  = 0,length_function = len,)
            docs = text_splitter.create_documents(data)
            #logging.debug("Documents created: %s", docs)
            print(docs)
            print(type(docs))
            
            
            embeddings = OpenAIEmbeddings(openai_api_key='sk-key')
            print(type(embeddings))
            print("Embeddings initialized.")
    
            print("Creating vectors from documents...")
            vectors = FAISS.from_documents(docs, embeddings)
            #print(embeddings.embed_query)
            print("Vectors created.")
            self.chain = ConversationalRetrievalChain.from_llm(
            llm=ChatOpenAI(temperature=0.0, model_name='gpt-3.5-turbo-16k', openai_api_key='sk-key'),
        retriever=vectors.as_retriever(), max_tokens_limit=16384, combine_docs_chain_kwargs={"prompt": QA_PROMPT}
            )

    
            print("Creating conversational retrieval chain...")
      
        except AttributeError as e:
            print("AttributeError occurred in OpenAI and FAISS part:", e)

<class 'pandas.core.frame.DataFrame'>
[Document(page_content='Q', metadata={}), Document(page_content='u', metadata={}), Document(page_content='e', metadata={}), Document(page_content='s', metadata={}), Document(page_content='t', metadata={}), Document(page_content='i', metadata={}), Document(page_content='o', metadata={}), Document(page_content='n', metadata={}), Document(page_content='A', metadata={}), Document(page_content='n', metadata={}), Document(page_content='s', metadata={}), Document(page_content='w', metadata={}), Document(page_content='e', metadata={}), Document(page_content='r', metadata={})]
<class 'list'>
<class 'langchain.embeddings.openai.OpenAIEmbeddings'>
Embeddings initialized.
Creating vectors from documents...
Vectors created.
Creating conversational retrieval chain...
Conversational retrieval chain created.
======== Running on http://0.0.0.0:3978 ========

核心问题 #

1. 数据格式不匹配：RecursiveCharacterTextSplitter.create_documents()要求传入文本字符串列表，但当前直接传入了DataFrame，导致分割器错误地将列名拆成单个字符（日志中可见单个字母的Document），完全未使用Excel中的实际问答数据。
2. 分割器参数不合理：chunk_size=1强制将所有文本拆分为单个字符，生成的向量无实际意义，模型无法检索到有效自定义数据。

修复步骤 #

1. 转换DataFrame为有效文本列表 #

将Excel中的每一行问答对拼接成完整文本，生成字符串列表：

# 新增：将DataFrame的问答行转换为文本列表
texts = []
for _, row in data.iterrows():
    # 假设最后两列分别是Question和Answer列
    question = row.iloc[0]
    answer = row.iloc[1]
    # 拼接成模型可识别的问答格式
    texts.append(f"问题：{question}\n答案：{answer}")

2. 调整文本分割器参数 #

设置合理的chunk_size和chunk_overlap，避免拆分过细：

# 修改分割器参数
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,  # 根据实际文本长度调整
    chunk_overlap=100,  # 保留上下文重叠，提升检索准确性
    length_function=len
)
docs = text_splitter.create_documents(texts)  # 传入处理好的文本列表

3. 验证生成的Documents #

修改后日志中应显示包含完整问答内容的Document，而非单个字符，此时FAISS生成的向量才能正确对应自定义数据，模型调用时即可检索到相关内容。

内容的提问来源于stack exchange，提问作者Ahmed Ewis