SeldonClient调用TensorFlow Serving gRPC接口返回StatusCode.UNIMPLEMENTED问题

模型签名信息 #

The given SavedModel SignatureDef contains the following input(s):
  inputs['Conv1_input'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, 28, 28, 1)
      name: serving_default_Conv1_input:0
The given SavedModel SignatureDef contains the following output(s):
  outputs['Dense'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, 10)
      name: StatefulPartitionedCall:0
Method name is: tensorflow/serving/predict

Dockerfile配置 #

FROM tensorflow/serving

ARG MODEL_PATH

# Define the model base path
ENV MODEL_BASE_PATH=/models

RUN mkdir -p $MODEL_BASE_PATH

# This will copy the model into the models/model dircetory in the container
COPY $MODEL_PATH /models/classifier
ENV MODEL_NAME=classifier

# REST PORT
EXPOSE 8500
# GRPC PORT
EXPOSE 8501

SeldonDeployment部署清单 #

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: tfserving
spec:
  annotations:
    seldon.io/executor: "true"
  protocol: tensorflow
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: tf-serve
          imagePullPolicy: Never
          name: model
          ports:
          - containerPort: 8501
            name: http
            protocol: TCP
          - containerPort: 8500
            name: grpc
            protocol: TCP
    graph:
      name: model
      type: MODEL
      endpoint:
        type: GRPC
        httpPort: 8501
        grpcPort: 8500
    name: template
    replicas: 1

直接调用TensorFlow Serving成功的代码 #

通过端口转发kubectl port-forward svc/tfserving-template-model -n seldon-services 8500:8500后，以下代码可正常获取预测结果：

MAX_MESSAGE_LENGTH = 2000000000
REQUEST_TIMEOUT = 90

class TfServing:
    def __init__(
        self, 
        host_port = "localhost:8500"
):
        channel = grpc.insecure_channel(
            host_port,
            options = [
                ("grpc.max_send_message_length", MAX_MESSAGE_LENGTH),
                ("grpc.max_receive_message_length", MAX_MESSAGE_LENGTH)
            ]
        )

        self.stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)
        self.req = predict_pb2.PredictRequest()
        self.req.model_spec.name = "classifier"

    def predict(self, image):
        tensor = tf.make_tensor_proto(image)
        self.req.inputs["Conv1_input"].CopyFrom(tensor)
        response = self.stub.Predict(self.req, REQUEST_TIMEOUT)
        output_tensor_proto = response.outputs["Dense"]
        shape  = tf.TensorShape(output_tensor_proto.tensor_shape)
        result = tf.reshape(output_tensor_proto.float_val, shape)
        return result.numpy()

if __name__ == "__main__":
    serving_model = TfServing()
    predictions = serving_model.predict(
        image = np.float32(
            np.uint8(
                np.random.random((1, 28, 28, 1)) * 255
            )
        )
    )

SeldonClient调用失败的情况 #

调用代码 #

sc = SeldonClient(
    deployment_name="tfserving", 
    namespace="seldon-services",
    gateway_endpoint="localhost:8500",
    grpc_max_send_message_length=20000000,
    grpc_max_receive_message_length=20000000,
)

r = sc.predict(
    gateway="seldon", 
    transport="grpc", 
    payload_type="tftensor",
    names = ["Conv1_input"],
    data=np.float32(
            np.uint8(
                np.random.random((1, 28, 28, 1)) * 255
            )
        ),
    
)

错误信息 #

Success:False message:<_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNIMPLEMENTED
        details = ""
        debug_error_string = "UNKNOWN:Error received from peer ipv6:%5B::1%5D:8500 {grpc_message:"", grpc_status:12, created_time:"2022-12-03T16:56:55.474244-06:00"}"

解决方案 #

1. 修正端口转发目标 #

当前直接转发模型的GRPC端口，但SeldonClient需要通过Seldon Gateway访问模型。转发Gateway的GRPC端口（默认9000）：

kubectl port-forward svc/seldon-gateway -n seldon-system 9000:9000

2. 调整SeldonClient初始化参数 #

• 网关端点改为转发后的localhost:9000
• 增大消息长度限制，与直接调用保持一致：

sc = SeldonClient(
    deployment_name="tfserving", 
    namespace="seldon-services",
    gateway_endpoint="localhost:9000",
    grpc_max_send_message_length=2000000000,
    grpc_max_receive_message_length=2000000000,
)

3. 完善predict调用参数 #

添加model_name指定模型名称"classifier"，匹配TensorFlow Serving的模型配置：

r = sc.predict(
    gateway="seldon", 
    transport="grpc", 
    payload_type="tftensor",
    names=["Conv1_input"],
    data=np.float32(np.uint8(np.random.random((1, 28, 28, 1)) * 255)),
    model_name="classifier"
)

4. 优化SeldonDeployment配置 #

移除graph.endpoint的显式配置，使用默认值即可，避免端口映射冲突：

graph:
  name: model
  type: MODEL
  # 移除以下endpoint配置
  # endpoint:
  #   type: GRPC
  #   httpPort: 8501
  #   grpcPort: 8500

内容的提问来源于stack exchange，提问作者Sam