调用AWS SageMaker GPT-2端点时持续超时错误求助

问题 #

调用AWS SageMaker部署的GPT-2端点生成内容时持续超时，已尝试增加实例数量、延长超时时间、增加重试次数，问题仍未解决。

代码 #

import os
import boto3
from botocore.config import Config

# Set AWS credentials using environment variables
os.environ["AWS_ACCESS_KEY_ID"] = "key"
os.environ["AWS_SECRET_ACCESS_KEY"] = "secret key"

# Configure the timeout settings and retry attempts for the Boto3 client
config = Config(
    read_timeout=300,  # Increase the read timeout to 300 seconds (5 minutes)
    connect_timeout=60,  # Increase the connect timeout to 60 seconds
    retries={'max_attempts': 5}  # Increase the maximum number of retry attempts to 5
)

sagemaker_client = boto3.client('sagemaker')
runtime_client = boto3.client('runtime.sagemaker', config=config)

# Replace 'gpt-2-model' with the actual name of your deployed GPT-2 model
gpt2_model_name = 'gpt-2-model'

# Replace 'gpt2-endpoint' with the actual name of your SageMaker endpoint
endpoint_name = 'gpt2-endpoint'

# Get information about the deployed GPT-2 model
try:
    gpt2_model_info = sagemaker_client.describe_model(ModelName=gpt2_model_name)
    print("GPT-2 Model Information:")
    print(gpt2_model_info)
except Exception as e:
    print(f"Error retrieving GPT-2 model information: {e}")

# Get endpoint information
try:
    endpoint_info = sagemaker_client.describe_endpoint(EndpointName=endpoint_name)
    endpoint_config_name = endpoint_info['EndpointConfigName']
    print("Endpoint Information:")
    print(endpoint_info)
except Exception as e:
    print(f"Error retrieving endpoint information: {e}")

def generate_content(prompt, max_length=500):
    try:
        response = runtime_client.invoke_endpoint(
            EndpointName=endpoint_name,
            ContentType='text/plain',
            Body=prompt.encode('utf-8'),
            Accept='text/plain'
        )
        generated_content = response['Body'].read().decode('utf-8')
        return generated_content[:max_length]
    except Exception as e:
        print(f"Error generating content: {e}")
        return None

# Test the generate_content function
prompt = "Write a blog post introduction about the benefits of automated marketing platforms."
generated_content = generate_content(prompt)

if generated_content:
    print("Generated Content:")
    print(generated_content)
else:
    print("Failed to generate content.")

错误输出 #

(env) macbook@Macbooks-MacBook-Pro content_generation % python content_generator.py
GPT-2 Model Information:
{'ModelName': 'gpt-2-model', 'Containers': [{'Image': '763104351884.dkr.ecr.eu-north-1.amazonaws.com/pytorch-inference:1.12.1-cpu-py38', 'Mode': 'SingleModel'}], 'ExecutionRoleArn': 'arn:aws:iam::471112657772:role/sagemaker', 'CreationTime': datetime.datetime(2024, 4, 19, 4, 38, 3, 872000, tzinfo=tzlocal()), 'ModelArn': 'arn:aws:sagemaker:eu-north-1:471112657772:model/gpt-2-model', 'EnableNetworkIsolation': False, 'DeploymentRecommendation': {'RecommendationStatus': 'COMPLETED', 'RealTimeInferenceRecommendations': [{'RecommendationId': 'gpt-2-model/pxjHziEz', 'InstanceType': 'ml.c5d.xlarge', 'Environment': {}}, {'RecommendationId': 'gpt-2-model/0TscHZCS', 'InstanceType': 'ml.g4dn.xlarge', 'Environment': {}}, {'RecommendationId': 'gpt-2-model/4pQJVxdM', 'InstanceType': 'ml.c5.2xlarge', 'Environment': {}}]}, 'ResponseMetadata': {'RequestId': 'bfeae9be-a458-4e89-baed-9d9954a9f1fe', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'bfeae9be-a458-4e89-baed-9d9954a9f1fe', 'content-type': 'application/x-amz-json-1.1', 'content-length': '731', 'date': 'Sun, 21 Apr 2024 07:27:19 GMT'}, 'RetryAttempts': 0}}
Endpoint Information:
{'EndpointName': 'gpt2-endpoint', 'EndpointArn': 'arn:aws:sagemaker:eu-north-1:471112657772:endpoint/gpt2-endpoint', 'EndpointConfigName': 'gpt2-config', 'ProductionVariants': [{'VariantName': 'default-variant-name', 'DeployedImages': [{'SpecifiedImage': '763104351884.dkr.ecr.eu-north-1.amazonaws.com/pytorch-inference:1.12.1-cpu-py38', 'ResolvedImage': '763104351884.dkr.ecr.eu-north-1.amazonaws.com/pytorch-inference@sha256:749b039a90e417706ed17cad5c977b5bc380d1dd2ed68b339e17a494ddc97b4c', 'ResolutionTime': datetime.datetime(2024, 4, 20, 14, 20, 57, 370000, tzinfo=tzlocal())}], 'CurrentWeight': 1.0, 'DesiredWeight': 1.0, 'CurrentInstanceCount': 2, 'DesiredInstanceCount': 2}], 'EndpointStatus': 'InService', 'CreationTime': datetime.datetime(2024, 4, 19, 10, 7, 28, 593000, tzinfo=tzlocal()), 'LastModifiedTime': datetime.datetime(2024, 4, 20, 14, 23, 20, 819000, tzinfo=tzlocal()), 'ResponseMetadata': {'RequestId': '858530d1-4e9b-458b-a298-bd1837479e25', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '858530d1-4e9b-458b-a298-bd1837479e25', 'content-type': 'application/x-amz-json-1.1', 'content-length': '704', 'date': 'Sun, 21 Apr 2024 07:27:19 GMT'}, 'RetryAttempts': 0}}
Error generating content: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (0) from primary with message "Your invocation timed out while waiting for a response from container primary. Review the latency metrics for each container in Amazon CloudWatch, resolve the issue, and try again.". See https://eu-north-1.console.aws.amazon.com/cloudwatch/home?region=eu-north-1#logEventViewer:group=/aws/sagemaker/Endpoints/gpt2-endpoint in account 471112657772 for more information.
Failed to generate content.

解决方案 #

1. 切换至GPU实例 #

当前使用的是CPU实例，GPT-2文本生成在CPU上速度极慢，是超时的核心原因。根据模型部署推荐，换成ml.g4dn.xlarge这类GPU实例：

• 登录SageMaker控制台，找到对应端点配置gpt2-config
• 修改生产变体的实例类型为ml.g4dn.xlarge，更新端点

2. 调整推理参数 #

先降低单次生成的文本长度，验证端点能正常返回结果后再逐步调高：

• 将generate_content函数中的max_length从500改为100测试
• 如果是自定义部署的模型，在推理代码中添加temperature、top_p等参数限制生成范围，避免无冗余生成

3. 配置容器超时 #

仅设置Boto3客户端超时不够，还需调整容器自身的超时限制：

• 使用官方PyTorch容器时，在端点配置中添加环境变量SAGEMAKER_MODEL_SERVER_TIMEOUT，设置值为300（单位：秒）
• 自定义容器的话，在服务脚本（如Flask）中设置响应超时参数

4. 查看容器日志定位问题 #

进入CloudWatch的SageMaker端点日志组，检查：

• 模型是否加载成功，是否有内存不足报错
• 推理请求的处理时长，确认是计算瓶颈还是其他异常

5. 优化请求格式 #

改用JSON格式传递请求，便于灵活控制生成参数：

import json

def generate_content(prompt, max_length=100):
    try:
        payload = json.dumps({
            "prompt": prompt,
            "max_length": max_length
        })
        response = runtime_client.invoke_endpoint(
            EndpointName=endpoint_name,
            ContentType='application/json',
            Body=payload,
            Accept='application/json'
        )
        generated_content = json.loads(response['Body'].read().decode('utf-8'))
        return generated_content.get('generated_text', '')[:max_length]
    except Exception as e:
        print(f"Error generating content: {e}")
        return None

同时确保模型容器的推理代码能解析JSON格式的请求。

内容的提问来源于stack exchange，提问作者Divine