GGML（Llama CPP）模型在Python中表现失常的问题求助

核心问题分析 #

• Python环境下模型不遵循指令、Shell中表现正常，本质是prompt模板差异：Shell版llama.cpp默认使用贴合模型训练的prompt格式，而Python绑定（如llama-cpp-python）或LangChain的默认prompt不符合模型预期格式。
• LangChain Agent调用失败，是因为模型输出不符合Agent要求的Action格式，且对工具名称识别错误。

针对性解决步骤 #

1. 统一Python环境的Prompt格式 #

GGML模型（如Manticore）通常遵循特定对话格式，比如Manticore的默认格式为：

### Instruction:
{user_query}

### Response:

在Python中直接调用时，必须手动封装prompt，而非传入原始问题：

def llm(query):
    prompt = f"### Instruction:\n{query}\n\n### Response:\n"
    return model(prompt, max_tokens=200)[0]['generated_text'].split("### Response:\n")[-1].strip()

测试验证示例：

问题：llm("Can you solve math questions?")
预期回复：Yes, I can help solve math questions. Please provide the problem you need assistance with.

2. 修正LangChain Agent的Prompt模板 #

LangChain的Zero-Shot Agent默认prompt不匹配Manticore格式，需自定义符合模型要求的prompt，强制模型输出正确Action格式：

from langchain.prompts import PromptTemplate

# 自定义符合Manticore格式的Agent prompt
agent_prompt = PromptTemplate(
    input_variables=["input", "tools", "tool_names", "agent_scratchpad"],
    template="""### Instruction:
Solve the following task as best you can. You have access to the following tools:

{tools}

Use the following format strictly:
Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, must be one of [{tool_names}]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question

Begin!
Question: {input}
{agent_scratchpad}"""
)

# 初始化Agent时使用自定义prompt
zero_shot_agent = initialize_agent(
    agent="zero-shot-react-description",
    tools=tools,
    llm=llm,
    verbose=True,
    max_iterations=3,
    agent_kwargs={"prompt": agent_prompt}
)

3. 调整模型生成参数 #

在Python调用时，增加输出格式约束：

• 设置stop=["###", "Thought:"]，防止模型输出多余内容
• 将temperature设为0.1~0.3（过低易僵化，过高会偏离指令）
• 启用prefix_match确保模型从正确位置开始生成

示例参数配置：

model = Llama(
    model_path="./manticore-13b-q4_0.ggmlv3.q4_0.bin",
    n_ctx=2048,
    temperature=0.2,
    stop=["###", "Thought:"],
    verbose=False
)

4. 工具名称标准化 #

LangChain的llm-math工具标准名称为Calculator，必须确保模型输出的Action严格为该名称，不能有变体（如"Regular Calculator"、"Phone Calculator"）。自定义prompt已明确限制工具名称范围，配合低temperature可减少模型发散输出。

验证效果 #

调整后运行Agent代码，模型应能正确输出：

Entering new AgentExecutor chain...
Thought: I need to use the Calculator to solve this math problem.
Action: Calculator
Action Input: (4.5*2.1)^2.2
Observation: 106.6832308753843
Thought: I now know the final answer
Final Answer: 106.6832308753843
> Finished chain.

内容的提问来源于stack exchange，提问作者Jawad Mansoor