循环中使用FAISS.from_documents加载分块TXT文件报错求助

将大体积文本拆分为1-86号TXT分块文件后，在for循环中调用FAISS.from_documents(docs, embeddings)持续报错，代码如下：

#export PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:120
#25 25 2 hr attempt

from langchain.document_loaders import TextLoader 
import textwrap
import os
import PyPDF2
from langchain.text_splitter import CharacterTextSplitter
import os
from langchain.embeddings import HuggingFaceEmbeddings 
from langchain.llms import HuggingFaceHub
import time
from langchain.document_loaders import TextLoader 
from langchain.vectorstores import FAISS
from langchain.chains.question_answering import load_qa_chain

faiss_objects = []  # Create an empty list to store FAISS objects
num = 4
queryText = "what is ISBN number"

for i in range(0,1):
    #------------------------------------------------------------------------------------------------------------------------------------------

    os.environ["HUGGINGFACEHUB_API_TOKEN"] = ' '

    db = []  # Initialize an empty list

    dir = "/home/r20/Downloads/Hugging_dace/pdf2txt/TXT/ 4.txt"
    v = "/home/r20/Downloads/Hugging_dace/pdf2txt/TXT/ " + str(i) + ".txt"

    var_name = f"variable_{i}"
    globals()[var_name] = 1

    loader = TextLoader(v)
    document = loader.load()

    text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
    docs = text_splitter.split_documents(document)

    embeddings = HuggingFaceEmbeddings()
    #print(embeddings)
    #print(FAISS.from_documents(docs, embeddings))

    llm= HuggingFaceHub(repo_id="google/flan-t5-xxl", model_kwargs={"temperature":0.8, "max_length":512})
    chain = load_qa_chain(llm, chain_type="stuff")

    globals()[var_name] = FAISS.add_item(docs, embeddings) 

'''docsResult = globals()[var_name].similarity_search(queryText)
print(chain.run(input_documents = docsResult, question = queryText))
time.sleep(2)
'''
#queryText = (str(input("TYPE YOUR QUERY  "))+ "if don't know just answer: 0")

print('done')

1. 错误的FAISS方法调用 #

FAISS没有add_item方法，正确操作：

• 首次创建索引用FAISS.from_documents(docs, embeddings)
• 往已有索引追加文档用faiss_index.add_documents(new_docs)

2. 循环范围错误 #

当前range(0,1)仅处理0号文件，要覆盖1-86号文件，需改为range(1, 87)

3. 全局变量滥用 #

用globals()存储FAISS对象易引发意外问题，直接用初始化的faiss_objects列表存储更安全可控

4. 资源重复初始化 #

embeddings、llm、chain无需在循环内重复创建，放到循环外可大幅节省系统资源

5. 文件路径错误 #

路径中TXT目录后多了空格，会导致文件找不到，需修正为"/home/r20/Downloads/Hugging_dace/pdf2txt/TXT/" + str(i) + ".txt"

6. 环境变量重复设置 #

HUGGINGFACEHUB_API_TOKEN只需设置一次，放到循环外即可

修正后的代码示例 #

#export PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:120
#25 25 2 hr attempt

from langchain.document_loaders import TextLoader 
import textwrap
import os
import PyPDF2
from langchain.text_splitter import CharacterTextSplitter
from langchain.embeddings import HuggingFaceEmbeddings 
from langchain.llms import HuggingFaceHub
import time
from langchain.vectorstores import FAISS
from langchain.chains.question_answering import load_qa_chain

# 全局配置提前设置
os.environ["HUGGINGFACEHUB_API_TOKEN"] = '你的API_TOKEN'
embeddings = HuggingFaceEmbeddings()
llm= HuggingFaceHub(repo_id="google/flan-t5-xxl", model_kwargs={"temperature":0.8, "max_length":512})
chain = load_qa_chain(llm, chain_type="stuff")

faiss_objects = []  # 存储每个分块的FAISS索引
queryText = "what is ISBN number"
txt_dir = "/home/r20/Downloads/Hugging_dace/pdf2txt/TXT/"

# 处理1-86号文件
for i in range(1, 87):
    file_path = txt_dir + str(i) + ".txt"
    
    # 加载并拆分文档
    loader = TextLoader(file_path)
    document = loader.load()
    text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
    docs = text_splitter.split_documents(document)
    
    # 创建FAISS索引并存储
    faiss_index = FAISS.from_documents(docs, embeddings)
    faiss_objects.append(faiss_index)
    
    # 可选：测试当前索引的查询
    docsResult = faiss_index.similarity_search(queryText)
    print(f"文件{i}查询结果：")
    print(chain.run(input_documents=docsResult, question=queryText))
    time.sleep(2)

# 可选：合并所有FAISS索引（如果需要统一查询）
# combined_index = faiss_objects[0]
# for index in faiss_objects[1:]:
#     combined_index.merge_from(index)

print('done')

内容的提问来源于stack exchange，提问作者ron