Gemini Pro 1.0调用报错Finish reason:2的解码与修复咨询

问题 #

使用以下基于gcloud认证的Python代码调用Vertex AI的Gemini Pro 1.0：

import vertexai
from vertexai.preview.generative_models import GenerativeModel, ChatSession

project_id = "<PROJECT_ID>"
location = "us-central1"

vertexai.init(project=project_id, location=location)

model = GenerativeModel("gemini-1.0-pro")
chat = model.start_chat()

def get_chat_response(chat: ChatSession, prompt: str):
    response = chat.send_message(prompt)
    return response.text

批量发送约4500条prompt时，发送到第150条左右持续出现无明确说明的错误，需要解读Finish reason: 2.并获取修复方法，错误日志如下：

File "/Users/l028/dev/reach-collective-ml-playground/hf_1_0/qualification_check/gemini.py", line 42, in get_chat_response
    response = chat.send_message(prompt)
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/l028/.pyenv/versions/playground_env/lib/python3.11/site-packages/vertexai/generative_models/_generative_models.py", line 727, in send_message
    return self._send_message(
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/l028/.pyenv/versions/playground_env/lib/python3.11/site-packages/vertexai/generative_models/_generative_models.py", line 822, in _send_message
    self._response_validator(
  File "/Users/l028/.pyenv/versions/playground_env/lib/python3.11/site-packages/vertexai/generative_models/_generative_models.py", line 658, in _validate_response
    raise ResponseValidationError(
vertexai.generative_models._generative_models.ResponseValidationError: The model response did not completed successfully.
Finish reason: 2.
Finish message: .
Safety ratings: [category: HARM_CATEGORY_HATE_SPEECH
probability: NEGLIGIBLE
, category: HARM_CATEGORY_DANGEROUS_CONTENT
probability: NEGLIGIBLE
, category: HARM_CATEGORY_HARASSMENT
probability: NEGLIGIBLE
, category: HARM_CATEGORY_SEXUALLY_EXPLICIT
probability: NEGLIGIBLE
].
To protect the integrity of the chat session, the request and response were not added to chat history.
To skip the response validation, specify `model.start_chat(response_validation=False)`.
Note that letting blocked or otherwise incomplete responses into chat history might lead to future interactions being blocked by the service.
2024-03-08 14:35:48 [error    ] Error in LLM request: The model response did not completed successfully.
Finish reason: 2.
Finish message: .
Safety ratings: [category: HARM_CATEGORY_HATE_SPEECH
probability: NEGLIGIBLE
, category: HARM_CATEGORY_DANGEROUS_CONTENT
probability: NEGLIGIBLE
, category: HARM_CATEGORY_HARASSMENT
probability: NEGLIGIBLE
, category: HARM_CATEGORY_SEXUALLY_EXPLICIT
probability: NEGLIGIBLE
].
To protect the integrity of the chat session, the request and response were not added to chat history.
To skip the response validation, specify `model.start_chat(response_validation=False)`.
Note that letting blocked or otherwise incomplete responses into chat history might lead to future interactions being blocked by the service.

Process finished with exit code 1

解答 #

1. Finish reason:2的含义 #

Finish reason的数值对应Vertex AI生成模型的响应终止原因，其中2代表模型因达到令牌输出上限而终止响应，并非安全政策拦截（安全评级均为NEGLIGIBLE可佐证）。批量请求中，随着对话历史累积（即使单条prompt不长），会话总令牌数会逐渐接近模型的上下文窗口限制，触发该终止逻辑。

2. 修复方法 #

• 定期重置会话历史：每处理一定数量的prompt后，重新创建ChatSession，避免会话令牌持续累积。例如每处理100条prompt就调用chat = model.start_chat()重置会话。
• 临时禁用响应验证：按照错误提示，调用model.start_chat(response_validation=False)跳过验证，允许不完整响应进入会话，但需注意这可能导致后续请求被拦截，仅适合临时调试场景。
• 优化prompt长度：缩短单条prompt的令牌数，降低会话历史的增长速度。
• 添加重试机制：针对ResponseValidationError捕获错误，若判定为令牌上限触发的终止，进行指数退避重试，重试前重置会话。

修改后的示例代码（包含会话重置和重试逻辑）：

import vertexai
from vertexai.preview.generative_models import GenerativeModel, ChatSession
from vertexai.generative_models._generative_models import ResponseValidationError
import time

project_id = "<PROJECT_ID>"
location = "us-central1"

vertexai.init(project=project_id, location=location)

model = GenerativeModel("gemini-1.0-pro")

def get_chat_response(chat: ChatSession, prompt: str, max_retries=3):
    for attempt in range(max_retries):
        try:
            response = chat.send_message(prompt)
            return response.text
        except ResponseValidationError as e:
            if "Finish reason: 2" in str(e):
                # 因令牌上限失败，重置会话并重试
                chat = model.start_chat()
                time.sleep(2 ** attempt)  # 指数退避等待
            else:
                raise
    raise Exception("Max retries exceeded")

# 批量处理示例
batch_prompts = [...]  # 替换为你的4500条prompt列表
chat = model.start_chat()
for idx, prompt in enumerate(batch_prompts):
    # 每100条重置一次会话
    if idx % 100 == 0:
        chat = model.start_chat()
    try:
        response = get_chat_response(chat, prompt)
        # 此处添加响应处理逻辑
    except Exception as e:
        # 记录错误信息
        print(f"Error processing prompt {idx}: {e}")

内容的提问来源于stack exchange，提问作者MickWise