在Amazon SageMaker部署预训练模型时Ping健康检查失败求助

错误现象 #

• 终端控制台报错：

The primary container for production variant default-variant-name did not pass the ping health check. Please check CloudWatch logs for this endpoint.

• CloudWatch日志记录：

169.254.178.2 - - [06/Oct/2022:05:43:42 +0000] "GET /ping HTTP/1.1" 200 0 "-" "AHC/2.0"

模型为pickle格式，已打包成tar.gz存储在S3中，相关代码如下：

serve.py代码 #

import json
import joblib
import numpy as np
from sklearn import svm
import os
import sklearn
import pickle
import boto3
import pickle
import tarfile

def init():

    global model
    s3_bucket = 'sagemaker-model-artifacts-dt'

    model_filename = 'svm-model.tar.gz'

    model_s3_key = model_filename

    model_url = f's3://{s3_bucket}/{model_s3_key}'
    print(model_url)
    my_tar = tarfile.open("svm-model.tar.gz")
    my_tar.extractall('./')
    model = pickle.load(open('svm-model.pkl','rb'))
    print(model)

def run(raw_data):
    # Get the input data as a numpy array
    print(raw_data)
    data = np.array(json.loads(raw_data)['data'])
#     data = scaler.transform(data)
    # Get a prediction from the model
    predictions: np.ndarray = model.predict(data)
    # Return the predictions as any JSON serializable format
    return {
        "predictions": predictions.tolist()
    }

Dockerfile代码 #

FROM python:latest

ENV PYTHONUNBUFFERED 1

RUN apt-get -y update && apt-get install -y --no-install-recommends \
         wget \
         python3 \
         nginx \
         ca-certificates \
    && rm -rf /var/lib/apt/lists/*

RUN wget https://bootstrap.pypa.io/get-pip.py && python3 get-pip.py && \
    pip install joblib numpy sklearn boto3 && \
        rm -rf /root/.cache

ENV PYTHONUNBUFFERED=TRUE
ENV PYTHONDONTWRITEBYTECODE=TRUE
ENV PATH="/opt/program:${PATH}"

COPY service_files /opt/program

WORKDIR /opt/program

ENTRYPOINT ["python","/opt/program/serve.py"]

问题分析与修复方案 #

核心问题 #

SageMaker容器要求服务必须响应/ping和/invocations两个HTTP端点：

• /ping用于健康检查，需返回200状态码
• /invocations处理预测请求

当前代码仅定义了init和run函数，未启动HTTP服务器处理请求，CloudWatch中的200响应实际来自容器内默认的Nginx页面，并非模型服务进程的响应，因此SageMaker判定健康检查失败。

修复步骤 #

1. 修改serve.py，添加HTTP服务支持
   使用Flask搭建符合SageMaker要求的HTTP服务，同时利用SageMaker自动下载模型到容器/opt/ml/model目录的特性，无需手动从S3拉取：

   import json
   import numpy as np
   import pickle
   import tarfile
   import os
   from flask import Flask, request, jsonify
   
   app = Flask(__name__)
   model = None
   
   def init():
       global model
       model_dir = '/opt/ml/model'
       model_path = os.path.join(model_dir, 'svm-model.pkl')
       
       # 若模型是tar.gz包，先解压
       if not os.path.exists(model_path):
           tar_path = os.path.join(model_dir, 'svm-model.tar.gz')
           tar = tarfile.open(tar_path, 'r:gz')
           tar.extractall(model_dir)
           tar.close()
       
       model = pickle.load(open(model_path, 'rb'))
       print("模型加载完成")
   
   @app.route('/ping', methods=['GET'])
   def ping():
       return '', 200
   
   @app.route('/invocations', methods=['POST'])
   def invocations():
       raw_data = request.data.decode('utf-8')
       data = np.array(json.loads(raw_data)['data'])
       predictions = model.predict(data)
       return jsonify({"predictions": predictions.tolist()})
   
   if __name__ == '__main__':
       init()
       app.run(host='0.0.0.0', port=8080)
   

2. 更新Dockerfile
   添加Flask依赖，并暴露SageMaker默认监听的8080端口：

   FROM python:latest
   
   ENV PYTHONUNBUFFERED 1
   
   RUN apt-get -y update && apt-get install -y --no-install-recommends \
            wget \
            python3 \
            nginx \
            ca-certificates \
       && rm -rf /var/lib/apt/lists/*
   
   RUN wget https://bootstrap.pypa.io/get-pip.py && python3 get-pip.py && \
       pip install joblib numpy sklearn boto3 flask && \
           rm -rf /root/.cache
   
   ENV PYTHONUNBUFFERED=TRUE
   ENV PYTHONDONTWRITEBYTECODE=TRUE
   ENV PATH="/opt/program:${PATH}"
   
   COPY service_files /opt/program
   
   WORKDIR /opt/program
   
   EXPOSE 8080
   ENTRYPOINT ["python","/opt/program/serve.py"]
   

3. 部署注意事项

   • 确保S3中的模型包路径正确，SageMaker会自动将其下载到容器内的/opt/ml/model目录
   • 容器必须监听8080端口，这是SageMaker与服务通信的默认端口

内容的提问来源于stack exchange，提问作者Gautam Damle