LangChain+llama-cpp-python长Prompt输出无效内容问题求助

问题描述 #

在服务器工作站上通过LangChain结合llama-cpp-python部署Llama3-70B模型服务，短Prompt（如question1、question2）可正常响应，但处理**小于8192 tokens的长Prompt（如question3）**时，模型输出无意义内容（例如“a a a the the the / / / <> .....”），该问题在8B模型上同样存在。相同Prompt在Ollama上可正常运行，未检索到同类问题案例。已尝试多种调整，均未解决问题。

运行代码 #

from langchain_community.llms import LlamaCpp
from langchain_core.prompts import PromptTemplate
from langchain_core.callbacks.manager import CallbackManager
from langchain_core.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from langchain_core.output_parsers import StrOutputParser

import multiprocessing

question_system = """
You know all knowledges of over the world. Answer the user's question.
"""

question1 = """
Question: Show me a rap battle between Stephen Colbert and John Oliver. And traslate it to french of your answer. Also show me english and french versions both. I think it is better that the battle is longer.

Answer:
"""

question2 = """
Question: Make longest questions that you can be. I want to prove your ability, so make 20 extremely long questions as you as possible.

Answer: 
"""

question3 = """
Question List:

1. Can a group of highly skilled and extensively trained experts in the field of artificial intelligence, with access to vast amounts of data and computational resources, successfully develop an AI system that can accurately predict and prevent all types of cyber attacks, including those that exploit zero-day vulnerabilities and use advanced evasion techniques?

2. Would it be possible for a team of experienced software developers, working together with a group of skilled security experts, to design and implement a secure and reliable operating system, one that is capable of protecting sensitive data and preventing malicious code from executing, even in the presence of sophisticated attacks and exploits?

3. Can a group of researchers and scientists, working together across multiple disciplines, including computer science, biology, chemistry, and physics, successfully develop a new generation of advanced nanotechnology-based systems, ones that are capable of operating at the molecular level, with precision and control, to perform complex tasks, such as medical imaging, drug delivery, and tissue engineering?

4. Would it be possible for a team of skilled engineers and technicians, working together with a group of experienced scientists and researchers, to design and develop a new generation of advanced spacecraft, ones that are capable of traveling to other planets and celestial bodies in our solar system, with precision and control, to perform complex tasks, such as planetary exploration, resource utilization, and astrobiology research?

5. Can a group of highly skilled and extensively trained experts in the field of cybersecurity, with access to vast amounts of data and computational resources, successfully develop an AI-powered system that can accurately detect and prevent all types of cyber attacks, including those that exploit zero-day vulnerabilities and use advanced evasion techniques?

6. Would it be possible for a team of experienced software developers, working together with a group of skilled security experts, to design and implement a secure and reliable operating system, one that is capable of protecting sensitive data and preventing malicious code from executing, even in the presence of sophisticated attacks and exploits?

7. Can a group of researchers and scientists, working together across multiple disciplines, including computer science, biology, chemistry, and physics, successfully develop a new generation of advanced nanotechnology-based systems, ones that are capable of operating at the molecular level, with precision and control, to perform complex tasks, such as medical imaging, drug delivery, and tissue engineering?

8. Would it be possible for a team of skilled engineers and technicians, working together with a group of experienced scientists and researchers, to design and develop a new generation of advanced spacecraft, ones that are capable of traveling to other planets and celestial bodies in our solar system, with precision and control, to perform complex tasks, such as planetary exploration, resource utilization, and astrobiology research?

9. Can a group of highly skilled and extensively trained experts in the field of artificial intelligence, with access to vast amounts of data and computational resources, successfully develop an AI-powered system that can accurately predict and prevent all types of natural disasters, including earthquakes, hurricanes, wildfires, and floods?

10. Would it be possible for a team of experienced software developers, working together with a group of skilled security experts, to design and implement a secure and reliable operating system, one that is capable of protecting sensitive data and preventing malicious code from executing, even in the presence of sophisticated attacks and exploits?

11. Can a group of researchers and scientists, working together across multiple disciplines, including computer science, biology, chemistry, and physics, successfully develop a new generation of advanced nanotechnology-based systems, ones that are capable of operating at the molecular level, with precision and control, to perform complex tasks, such as medical imaging, drug delivery, and tissue engineering?

12. Would it be possible for a team of skilled engineers and technicians, working together with a group of experienced scientists and researchers, to design and develop a new generation of advanced spacecraft, ones that are capable of traveling to other planets and celestial bodies in our solar system, with precision and control, to perform complex tasks, such as planetary exploration, resource utilization, medical imaging, drug delivery, and asteroid

13. Can a group of highly skilled and extensively trained experts in the field of artificial intelligence, with access to vast amounts of data and computational resources, successfully develop an AI system that can accurately predict and prevent all types of cyber attacks, including those that exploit zero-day vulnerabilities and use advanced evasion techniques?

14. Would it be possible for a team of experienced software developers, working together with a group of skilled security experts, to design and implement a secure and reliable operating system, one that is capable of protecting sensitive data and preventing malicious code from executing, even in the presence of sophisticated attacks and exploits?

15. Can a group of researchers and scientists, working together across multiple disciplines, including computer science, biology, chemistry, and physics, successfully develop a new generation of advanced nanotechnology-based systems, ones that are capable of operating at the molecular level, with precision and control, to perform complex tasks, such as medical imaging, drug delivery, and tissue engineering?

16. Would it be possible for a team of skilled engineers and technicians, working together with a group of experienced scientists and researchers, to design and develop a new generation of advanced spacecraft, ones that are capable of traveling to other planets and celestial bodies in our solar system, with precision and control, to perform complex tasks, such as planetary exploration, resource utilization, and astrobiology research?

17. Can a group of highly skilled and extensively trained experts in the field of cybersecurity, with access to vast amounts of data and computational resources, successfully develop an AI-powered system that can accurately detect and prevent all types of cyber attacks, including those that exploit zero-day vulnerabilities and use advanced evasion techniques?

18. Would it be possible for a team of experienced software developers, working together with a group of skilled security experts, to design and implement a secure and reliable operating system, one that is capable of protecting sensitive data and preventing malicious code from executing, even in the presence of sophisticated attacks and exploits?

19. Can a group of researchers and scientists, working together across multiple disciplines, including computer science, biology, chemistry, and physics, successfully develop a new generation of advanced nanotechnology-based systems, ones that are capable of operating at the molecular level, with precision and control, to perform complex tasks, such as medical imaging, drug delivery, and tissue engineering?

20. Would it be possible for a team of skilled engineers and technicians, working together with a group of experienced scientists and researchers, to design and develop a new generation of advanced spacecraft, ones that are capable of traveling to other planets and celestial bodies in our solar system, with precision and control, to perform complex tasks, such as planetary exploration, resource utilization, and astrobiology research?

21. Can a group of highly skilled and extensively trained experts in the field of artificial intelligence, with access to vast amounts of data and computational resources, successfully develop an AI-powered system that can accurately predict and prevent all types of natural disasters, including earthquakes, hurricanes, wildfires, and floods?

22. Would it be possible for a team of experienced software developers, working together with a group of skilled security experts, to design and implement a secure and reliable operating system, one that is capable of protecting sensitive data and preventing malicious code from executing, even in the presence of sophisticated attacks and exploits?

23. Can a group of researchers and scientists, working together across multiple disciplines, including computer science, biology, chemistry, and physics, successfully develop a new generation of advanced nanotechnology-based systems, ones that are capable of operating at the molecular level, with precision and control, to perform complex tasks, such as medical imaging, drug delivery, and tissue engineering?

24. Would it be possible for a team of skilled engineers and technicians, working together with a group of experienced scientists and researchers, to design and develop a new generation of advanced spacecraft, ones that are capable of traveling to other planets and celestial bodies in our solar system, with precision and control, to perform complex tasks, such as planetary exploration, resource utilization, medical imaging, drug delivery, and asteroid?

Question: Answer about the above questions shortly, and count the questions.

Answer: 
"""

template = """
{System}<|start_header_id|>system<|end_header_id|>

{System}<|eot_id|>

{Prompt}<|start_header_id|>user<|end_header_id|>

{Prompt}<|eot_id|>

<|start_header_id|>assistant<|end_header_id|>

{Response}<|eot_id|>
"""

prompt = PromptTemplate.from_template(template)

llm = LlamaCpp(
    model_path="/HDD/llama3/Meta-Llama-3-70B-Instruct.Q4_K_M.gguf",
    n_gpu_layers=-1,
    n_ctx=8192,
    n_batch=128,
    max_tokens=1024,
    callback_manager=CallbackManager([StreamingStdOutCallbackHandler()]),
    verbose=True,
    stop=["<|start_header_id|>", "<|end_header_id|>", "<|eot_id|>"],
    n_threads=multiprocessing.cpu_count()-1,
)

chain = prompt | llm | StrOutputParser()
res = chain.invoke({"System":question_system, "Prompt":question3, "Response":None})
print(res)

工作站配置 #

• 操作系统：Ubuntu 20.04
• GPU：6块RTX 8000（单卡VRAM 48GB）
• langchain版本：v0.2.8
• llama-cpp-python版本：v0.2.82
• 模型文件：Meta-Llama-3-70B-Instruct.Q4_K_M.gguf

已尝试的解决方案 #

• 调整配置参数：将n_ctx增至819210、n_batch增至12810，降低n_gpu_layers至1、2、4，问题依旧。
• 移除n_threads参数，无改善。
• 改用ChatLlamaCpp，问题仍存在。
• 相同Prompt在Ollama上可正常运行，无法定位原因。

内容的提问来源于stack exchange，提问作者bibiibibin