使用llama_index搭配本地Vicuna与嵌入模型时出现整数除零OSError

问题背景 #

尝试用LlamaIndex搭建简单信息检索系统，本地运行嵌入模型和LLM，但执行代码时始终抛出OSError: exception: integer divide by zero错误，程序直接崩溃，无法得到预期结果（如8 September 2022）。

代码实现 #

# Various imports here

max_input_size = 240
num_output = 120
max_chunk_overlap = 0
prompt_helper = PromptHelper(max_input_size, num_output, max_chunk_overlap)

embed_model = LangchainEmbedding(HuggingFaceEmbeddings(model_name="sentence-transformers/all-mpnet-base-v2"))

llm_predictor = LLMPredictor(llm=LlamaCpp(model_path="./Wizard-Vicuna-7B-Uncensored.ggmlv3.q4_0.bin", n_threads=2, n_gpu_layers=32, temperature=0.25, verbose=True))

service_context = ServiceContext.from_defaults(chunk_size_limit=512, llm_predictor=llm_predictor, embed_model=embed_model, prompt_helper=prompt_helper)
set_global_service_context(service_context)

inpt = "Give an answer to the following question as compact as possible. When did the Queen die?"
documents = SimpleDirectoryReader('./documents/').load_data()

index = GPTVectorStoreIndex.from_documents(documents, service_context=service_context)

index.storage_context.persist(persist_dir="./storage")

query_engine = index.as_query_engine(streaming=True, similarity_top_k=3)
response = query_engine.query(inpt)

response.print_response_stream()

文档内容 #

./documents/文件夹下包含两个文本文件：

Doc1.txt #

Wikipedia entry about the Queen of england (Queen Elisabeth II). From 1952 until her death in 2022, Elizabeth II of the House of Windsor was Queen of the United Kingdom of Great Britain and Northern Ireland and of 14 other sovereign states known as Commonwealth Realms, including their territories and dependent areas.
Born: April 21, 1926, Bruton Street, London, United Kingdom
Deceased: 8 September 2022, Balmoral Castle, United Kingdom

Doc2.txt #

Freddie Mercury was a British musician and one of the most important rock singers of the 1970s and 1980s. He is best known as the co-founder, composer and lead singer of the band Queen. Mercury composed world hits like Bohemian Rhapsody and We Are the Champions.
Born: September 5, 1946, Stone Town, Tanzania
Deceased: 24 November 1991, Kensington, London, United Kingdom

错误日志 #

llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 512
llama_model_load_internal: n_embd     = 4096
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 32
llama_model_load_internal: n_layer    = 32
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: n_ff       = 11008
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size =    0.07 MB
llama_model_load_internal: mem required  = 1932.71 MB (+ 1026.00 MB per state)
llama_model_load_internal: [cublas] offloading 32 layers to GPU
llama_model_load_internal: [cublas] total VRAM used: 3475 MB
..................................................................................................
llama_init_from_file: kv self size  =  256.00 MB
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | 

llama_print_timings:        load time =  1115.99 ms
llama_print_timings:      sample time =     6.59 ms /    21 runs   (    0.31 ms per token)
llama_print_timings: prompt eval time =  3579.16 ms /    80 tokens (   44.74 ms per token)
llama_print_timings:        eval time =  2160.82 ms /    21 runs   (  102.90 ms per token)
llama_print_timings:       total time = 10107.78 ms
Llama.generate: prefix-match hit
Llama.generate: prefix-match hit
Llama.generate: prefix-match hit
Exception in thread Thread-4 (_predict):
Traceback (most recent call last):
  File "./env\lib\threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "./env\lib\threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "./env\lib\site-packages\llama_index\llm_predictor\base.py", line 195, in _predict
    llm_prediction = retry_on_exceptions_with_backoff(
  File "./env\lib\site-packages\llama_index\utils.py", line 177, in retry_on_exceptions_with_backoff
    return lambda_fn()
  File "./env\lib\site-packages\llama_index\llm_predictor\base.py", line 196, in <lambda>
    lambda: llm_chain.predict(**full_prompt_args),
  File "./env\lib\site-packages\langchain\chains\llm.py", line 213, in predict
    return self(kwargs, callbacks=callbacks)[self.output_key]
  File "./env\lib\site-packages\langchain\chains\base.py", line 140, in __call__
    raise e
  File "./env\lib\site-packages\langchain\chains\base.py", line 134, in __call__
    self._call(inputs, run_manager=run_manager)
  File "./env\lib\site-packages\langchain\chains\llm.py", line 69, in _call
    response = self.generate([inputs], run_manager=run_manager)
  File "./env\lib\site-packages\langchain\chains\llm.py", line 79, in generate
    return self.llm.generate_prompt(
  File "./env\lib\site-packages\langchain\llms\base.py", line 134, in generate_prompt
    return self.generate(prompt_strings, stop=stop, callbacks=callbacks)
  File "./env\lib\site-packages\langchain\llms\base.py", line 191, in generate
    raise e
  File "./env\lib\site-packages\langchain\llms\base.py", line 185, in generate
    self._generate(prompts, stop=stop, run_manager=run_manager)
  File "./env\lib\site-packages\langchain\llms\base.py", line 436, in _generate
    self._call(prompt, stop=stop, run_manager=run_manager)
  File "./env\lib\site-packages\langchain\llms\llamacpp.py", line 225, in _call
    for token in self.stream(prompt=prompt, stop=stop, run_manager=run_manager):
  File "./env\lib\site-packages\langchain\llms\llamacpp.py", line 274, in stream
    for chunk in result:
  File "./env\lib\site-packages\llama_cpp\llama.py", line 761, in _create_completion
    for token in self.generate(
  File "./env\lib\site-packages\llama_cpp\llama.py", line 596, in generate
    self.eval(tokens)
  File "./env\lib\site-packages\llama_cpp\llama.py", line 325, in eval
    return_code = llama_cpp.llama_eval(
  File "./env\lib\site-packages\llama_cpp\llama_cpp.py", line 439, in llama_eval
    return _lib.llama_eval(ctx, tokens, n_tokens, n_past, n_threads)
OSError: exception: integer divide by zero

已尝试的排查措施 #

• 将流式输出改为streaming=False并直接打印结果，问题仍未解决
• 确认24GB显存充足且使用率极低，排除显存不足因素

内容的提问来源于stack exchange，提问作者Jan-Ole K.