使用LangChain的load_qa_chain时遇Token序列长度超出模型限制错误

错误信息 #

Token indices sequence length is longer than the specified maximum sequence length for this model (2108 > 1024

相关代码 #

from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

model="ehartford/WizardLM-7B-Uncensored"
text_gen_pipeline = pipeline(
    model = model, 
    model_kwargs= {
        "device_map": "auto", 
        "load_in_8bit": True, 
        # default-explain-code settings from https://platform.openai.com/examples
        "temperature": 0,
        "top_p": 1.0,
    },
    max_new_tokens=2500)
    
from langchain import HuggingFacePipeline
llm = HuggingFacePipeline(pipeline=text_gen_pipeline)

from langchain.text_splitter import RecursiveCharacterTextSplitter
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size = 1000,
    chunk_overlap = 50)
# data is loaded with GitLoader
chunks = text_splitter.split_documents(data)

from langchain.embeddings import HuggingFaceEmbeddings
embeddings = HuggingFaceEmbeddings()
from langchain.vectorstores.faiss import FAISS

vectorstore = FAISS.from_documents(chunks, embeddings)

query = "How do you use the HamburgerMenu component?"
docs = vectorstore.similarity_search(query)

from langchain.chains.question_answering import load_qa_chain
chain = load_qa_chain(llm, chain_type="stuff")

result = chain.run(input_documents=docs, question=query)

问题原因 #

WizardLM-7B-Uncensored模型的最大上下文窗口为1024 token，load_qa_chain的stuff模式会将所有检索到的文档拼接后塞入prompt，加上问题和内置提示词的token，总长度超过了模型限制。即便使用map_reduce模式，若单块文档+子任务提示词的长度超限，也会触发该错误。

解决方案 #

• 按token数分割文本：RecursiveCharacterTextSplitter默认按字符数分割，和token数不匹配。改用模型的tokenizer计算长度，将chunk_size设为小于模型上限的值（比如800，留足提示词空间）：

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("ehartford/WizardLM-7B-Uncensored")

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size = 800,
    chunk_overlap = 50,
    length_function = lambda x: len(tokenizer.encode(x)),
    separators=["\n\n", "\n", " ", ""]
)
chunks = text_splitter.split_documents(data)

• 减少检索文档数量：限制similarity_search返回的文档数，避免拼接后长度超限：

docs = vectorstore.similarity_search(query, k=3)  # 根据实际场景调整k值

• 明确设置模型最大输入长度：创建pipeline时给模型加上max_length参数，强制截断过长输入：

text_gen_pipeline = pipeline(
    model = model, 
    model_kwargs= {
        "device_map": "auto", 
        "load_in_8bit": True, 
        "temperature": 0,
        "top_p": 1.0,
        "max_length": 1024  # 匹配模型的最大token限制
    },
    max_new_tokens=2500)

• 更换长上下文模型：若业务需要处理更长文本，可改用支持更大上下文窗口的模型（比如WizardLM-7B-16k-Uncensored），但会增加显存消耗。

内容的提问来源于stack exchange，提问作者U Avalos