使用TruLens评估Falcon 7B时出现响应截断问题

问题场景 #

使用LangChain定义Prompt模板、构建LLM链，结合trulens_eval的HuggingFace模块检测响应毒性时，调用TruChain返回的LLM响应内容被截断，无法生成完整段落。

原始代码 #

from langchain import PromptTemplate
from langchain.chains import  LLMChain
from langchain.prompts.chat import (ChatPromptTemplate,HumanMessagePromptTemplate)
from langchain import HuggingFaceHub
from langchain.chat_models import ChatOpenAI
from trulens_eval import TruChain

full_prompt = HumanMessagePromptTemplate(
    prompt=PromptTemplate(
        template="Please provide detailed helpful response with relevant background information for the following: {prompt}. Provide a complete paragraph of the response",
            input_variables=["prompt"],
        )
    )
chat_prompt_template = ChatPromptTemplate.from_messages([full_prompt])
model = HuggingFaceHub(repo_id='tiiuae/falcon-7b-instruct', model_kwargs={"temperature":0.5})
chain = LLMChain(llm=model, prompt=chat_prompt_template)

from trulens_eval import Feedback, Huggingface, Query
hugs=Huggingface()
f_toxicity=Feedback(hugs.not_toxic).on(text=Query.RecordOutput)

truchain=TruChain(chain,app_id="testapp_validation",feedbacks=[f_toxicity])
llm_response3=truchain("What is Machine Learning and Artificial Intelligence")
display(llm_response3)

截断的输出 #

{'prompt': 'What is Machine Learning and Artificial Intelligence',
 'text': 'Machine learning is the process of learning to do things by analyzing data and incorporating it into'}

问题原因 #

使用HuggingFaceHub调用falcon-7b-instruct模型时，未在model_kwargs中设置max_new_tokens参数，模型默认的生成最大token数较小，导致生成内容被提前截断。

解决方案 #

在初始化HuggingFaceHub时，添加max_new_tokens参数，设置足够大的值以支持生成完整段落（比如设置为512或1024，可根据需求调整）。

修改后的代码 #

from langchain import PromptTemplate
from langchain.chains import  LLMChain
from langchain.prompts.chat import (ChatPromptTemplate,HumanMessagePromptTemplate)
from langchain import HuggingFaceHub
from langchain.chat_models import ChatOpenAI
from trulens_eval import TruChain

full_prompt = HumanMessagePromptTemplate(
    prompt=PromptTemplate(
        template="Please provide detailed helpful response with relevant background information for the following: {prompt}. Provide a complete paragraph of the response",
            input_variables=["prompt"],
        )
    )
chat_prompt_template = ChatPromptTemplate.from_messages([full_prompt])
# 添加max_new_tokens参数，调整生成的最大token数
model = HuggingFaceHub(repo_id='tiiuae/falcon-7b-instruct', model_kwargs={"temperature":0.5, "max_new_tokens": 512})
chain = LLMChain(llm=model, prompt=chat_prompt_template)

from trulens_eval import Feedback, Huggingface, Query
hugs=Huggingface()
f_toxicity=Feedback(hugs.not_toxic).on(text=Query.RecordOutput)

truchain=TruChain(chain,app_id="testapp_validation",feedbacks=[f_toxicity])
llm_response3=truchain("What is Machine Learning and Artificial Intelligence")
display(llm_response3)

补充说明 #

• max_new_tokens的取值需根据实际需求和模型能力调整，过大可能导致生成时间变长或出现无意义内容；
• 部分模型可能还支持max_length参数，但max_new_tokens更直观，指定的是新增生成的token数，而非总token数。

内容的提问来源于stack exchange，提问作者RAUNAK GHOSH