如何在ChromaDB中避免重复生成嵌入：仅按需执行

要实现仅在首次运行或嵌入目录为空时生成嵌入，核心思路是先检查Chroma持久化目录是否已有有效向量数据，再决定是加载现有库还是重新生成嵌入。具体实现如下：

修改后的代码 #

import os
from langchain_community.document_loaders import TextLoader
from langchain_community.vectorstores import Chroma
from langchain_community import embeddings
from langchain_community.chat_models import ChatOllama
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain.text_splitter import CharacterTextSplitter

model_local = ChatOllama(model="codellama:7b")
ollama_embeddings = embeddings.ollama.OllamaEmbeddings(model="nomic-embed-text")
persist_dir = "./vector/my_data"

# 检查持久化目录是否存在且有有效数据
vectorstore = None
if os.path.exists(persist_dir) and len(os.listdir(persist_dir)) > 0:
    # 加载已有的向量库
    vectorstore = Chroma(
        persist_directory=persist_dir,
        embedding_function=ollama_embeddings
    )
else:
    # 目录为空或不存在，重新生成嵌入
    loader = TextLoader("remedy.txt")
    raw_doc = loader.load()
    
    # 分割文档
    text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
    splitted_docs = text_splitter.split_documents(raw_doc)
    
    # 生成嵌入并存储到Chroma
    vectorstore = Chroma.from_documents(
        documents=splitted_docs,
        embedding=ollama_embeddings,
        persist_directory=persist_dir,
    )

retriever = vectorstore.as_retriever()

# RAG问答链
print("After RAG\n")
after_rag_template = """
    Answer the question based only on the following context:
    {context}
    Question {question}?
"""
after_rag_prompt = ChatPromptTemplate.from_template(after_rag_template)
after_rag_chain = (
    {"context": retriever, "question": RunnablePassthrough()}
    | after_rag_prompt
    | model_local
    | StrOutputParser()
)
print(after_rag_chain.invoke("What are Home Remedy for Common Cold?"))

关键改动说明 #

1. 目录检查逻辑：通过os.path.exists和os.listdir判断持久化目录是否存在且非空，Chroma初始化后会在目录下生成chroma.sqlite3和embeddings文件夹，只要目录有内容就认为已有有效向量数据。
2. 分支执行逻辑：
   • 若目录有效，直接加载现有Chroma向量库，跳过嵌入生成步骤；
   • 若目录不存在或为空，才执行文档加载、分割、嵌入生成和存储的流程。
3. 代码优化：统一了嵌入函数的命名，移除了未使用的导入，让代码更简洁。

内容的提问来源于stack exchange，提问作者raju