LangChain Agent邮件发送工具未遵循规则的问题排查与优化咨询

问题描述 #

我正在用LangChain、langchain_openai和一个自定义邮件发送工具开发智能体，这个工具叫mail_tool，设计时要求严格遵守规则和特定输入格式，但现在遇到了问题：智能体有时候不遵守规则，会编造邮箱地址，或者无法正确发送内容。

工具规则与输入格式 #

这个mail_tool必须严格遵守以下规则：

• 工具使用场景：只有当用户要求通过邮件发送信息，并且提供了自己的邮箱地址和所需内容时，才能使用这个工具。
• 禁止编造邮箱：绝对不能编造邮箱地址，只能使用用户提供的邮箱。
• 输入格式要求：输入必须是一个字符串形式的字典（因为模型只会返回字符串），格式如下：

{
    'mail': 'user_email',  # 用户的邮箱地址
    'content': 'content_to_send'  # 要发送的内容
}

相关代码实现 #

以下是我目前的实现代码：

import asyncio
from langchain_openai import ChatOpenAI
from langchain.tools import Tool
from langgraph.checkpoint.memory import MemorySaver
from langchain_core.messages import HumanMessage, AIMessage
from langgraph.graph import StateGraph, START, END, MessagesState
from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain.agents import create_openai_functions_agent, AgentExecutor
import json

# Model and tools configuration
model = ChatOpenAI(model="gpt-4", openai_api_key="KEY_HERE")

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a friendly assistant named Axel, and you will answer questions. You only provide answers of a maximum of 25 words. You respond in Spanish"),
    MessagesPlaceholder(variable_name="chat_history"),
    ("human", "{input}"),
    MessagesPlaceholder(variable_name="agent_scratchpad")
])

def send_mail_tool(data: str):
    try:
        parsed_data = json.loads(data)
        mail = parsed_data.get("mail")
        content = parsed_data.get("content")
        if mail and content:
            return "Email sent"
        else:
            return "Error: 'mail' or 'content' is missing"
    except Exception as e:
        return f"Error: {str(e)}"

mail_tool = Tool(
    name="send_mail_tool",
    func=send_mail_tool,
    description="""
        FOLLOW THESE RULES STRICTLY TO USE THIS TOOL:
        - Use this tool only when the user requests to send information via email and provides their email address and the required content.
        - Only use it for this purpose.
        - Do not invent email addresses; only use the one provided by the user.
        - Always, without exception, use the Input format.
        - this is the Input Format:
            The input must be a dictionary with the following keys:
            {
                'mail': 'user_email',  # The user's email address.
                'content': 'content_to_send'  # The information to be sent.
            }
    """
)

tools = [mail_tool]

agent = create_openai_functions_agent(llm=model, prompt=prompt, tools=tools)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=False)

workflow = StateGraph(state_schema=MessagesState)

async def call_agent(state: MessagesState):
    user_input = state["messages"][-1].content
    chat_history = state["messages"][:-1]
    response = await agent_executor.ainvoke({
        "input": user_input,
        "chat_history": chat_history
    })
    return {"messages": state["messages"] + [AIMessage(content=response["output"])]}

workflow.add_node("agent", call_agent)
workflow.add_edge(START, "agent")

memory = MemorySaver()
app = workflow.compile(checkpointer=memory)

config = {"configurable": {"thread_id": "user1234"}}

app.update_state(config, {"messages": [
    AIMessage(content="You are a friendly assistant named Axel, and you will answer questions. You only provide answers of a maximum of 25 words. You respond in Spanish")
]})

async def main():
    print("Assistant Online. Type 'exit' to quit.")
    while True:
        user_input = input("\nYou: ")
        if user_input.lower() == "exit":
            print("Goodbye!")
            break
        app.update_state(config, {"messages": [HumanMessage(content=user_input)]})
        state = app.get_state(config).values
        try:
            output = await app.ainvoke(state, config)
        except Exception as e:
            print("\nAssistant: Sorry, an error occurred while processing your request.")
            print(f"Details: {str(e)}")
            continue
        print("\nAssistant:")
        print(output["messages"][-1].content)

if __name__ == "__main__":
    asyncio.run(main())

具体问题表现 #

智能体有时候不遵守mail_tool的规则，具体问题如下：

• 编造邮箱地址：偶尔会自己生成邮箱地址，而不是使用用户提供的。
• 内容发送异常：有时候无法正确发送内容，或者不遵守指定的输入格式。

我的疑问 #

我该如何修复这种行为，确保智能体严格遵守mail_tool的规则和输入格式？有没有其他改进方法或者替代方案能让智能体的表现更稳定？

补充信息 #

• 我使用的语言模型是gpt-4。
• send_mail_tool只是一个模拟工具，不会真正发送邮件，但需要遵循指定的格式和规则。

备注：内容来源于stack exchange，提问作者Alexander idid