API Gateway调用Lambda触发SageMaker Endpoint报500错误求助

我的SageMaker终端节点通过Lambda直接调用时运行正常，但使用相同负载通过API Gateway REST API（已启用Lambda API集成的ANY方法部署）调用时，出现ModelError。

Lambda函数代码片段 #

@app.post("/dept_predictor")
@tracer.capture_method
def dept_predictor():
    # post_data: dict = app.current_event.json_body
    # try:
        print("Received event: " + json.dumps( app.current_event.body, indent=2))
        payload = app.current_event.body
        print(payload)
        # raise Exception('Malformed input ...')
    
        response  = runtime_client.invoke_endpoint(EndpointName=ENDPOINT_NAME, 
                                                  ContentType='application/json', 
                                                  Body=json.dumps(payload),
                                                  Accept='Accept')
        print(response)

API Gateway调用后的CloudWatch日志 #

[ERROR] ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (500) from model with message "<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>

SageMaker终端节点的CloudWatch日志 #

2023-01-19T00:14:05.961+05:30   Loading the hash vectorizer model ...

2023-01-19T00:14:05.961+05:30   2023-01-18 18:44:05,959 ERROR - sklearn_functions - Exception on /invocations [POST]

2023-01-19T00:14:05.961+05:30   Traceback (most recent call last): File "/miniconda3/lib/python3.7/site-packages/sagemaker_containers/_functions.py", line 93, in wrapper return fn(*args, **kwargs) File "/opt/ml/code/sklearn_functions.py", line 130, in input_fn pname = json.loads(request_body)["product_name"]

2023-01-19T00:14:05.962+05:30   TypeError: string indices must be integers

2023-01-19T00:14:05.962+05:30   During handling of the above exception, another exception occurred:

2023-01-19T00:14:05.962+05:30   Traceback (most recent call last): File "/miniconda3/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app response = self.full_dispatch_request() File "/miniconda3/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request rv = self.handle_user_exception(e) File "/miniconda3/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception reraise(exc_type, exc_value, tb) File "/miniconda3/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise raise value File "/miniconda3/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request rv = self.dispatch_request() File "/miniconda3/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/miniconda3/lib/python3.7/site-packages/sagemaker_containers/_transformer.py", line 200, in transform self._model, request.content, request.content_type, request.accept File "/miniconda3/lib/python3.7/site-packages/sagemaker_containers/_transformer.py", line 227, in _default_transform_fn data = self._input_fn(content, content_type) File "/miniconda3/lib/python3.7/site-packages/sagemaker_containers/_functions.py", line 95, in wrapper six.reraise(error_class, error_class(e), sys.exc_info()[2]) File "/miniconda3/lib/python3.7/site-packages/six.py", line 702, in reraise raise value.with_traceback(tb) File "/miniconda3/lib/python3.7/site-packages/sagemaker_containers/_functions.py", line 93, in wrapper return fn(*args, **kwargs) File "/opt/ml/code/sklearn_functions.py", line 130, in input_fn pname = json.loads(request_body)["product_name"]

2023-01-19T00:14:05.962+05:30   10.32.0.2 - - [18/Jan/2023:18:44:05 +0000] "POST /invocations HTTP/1.1" 500 290 "-" "AHC/2.0"

2023-01-19T00:14:05.962+05:30   sagemaker_containers._errors.ClientError: string indices must be integers

从SageMaker日志的TypeError: string indices must be integers可以看出，json.loads(request_body)返回的是字符串而非字典，导致无法通过["product_name"]取值，核心问题是API Gateway传递给Lambda的payload格式和直接调用Lambda时不一致，具体排查点：

• 修正Lambda的payload解析逻辑
  直接调用Lambda时，app.current_event.body可能是已解析的字典，但API Gateway调用时，body是原始JSON字符串。需先将字符串解析为字典再序列化传递给SageMaker：

  import json
  # ...
  payload = app.current_event.body
  # 检查是否为字符串，是则解析
  if isinstance(payload, str):
      payload = json.loads(payload)
  # 或者直接使用已封装的json_body
  # payload = app.current_event.json_body
  

• 检查API Gateway集成配置

  • 确认是否启用代理集成：非代理集成会对请求体进行额外包装，导致Lambda收到的payload结构异常；代理集成会直接传递原始请求体。
  • 验证映射模板：如果使用自定义映射模板，需确保输出是正确的JSON结构，避免嵌套字符串问题。

• 验证最终传递给SageMaker的Body
  在Lambda中打印json.dumps(payload)的结果，对比直接调用和API Gateway调用时的输出。若API Gateway调用时输出是嵌套的字符串（如"\"{\\\"product_name\\\": \\\"xxx\\\"}\""），说明存在重复序列化，需先解析原始字符串再处理。

• 检查SageMaker端点的input_fn
  确认input_fn中处理application/json请求的逻辑是否正确，确保请求体解析后能得到预期的字典结构。

内容的提问来源于stack exchange，提问作者iamabhaykmr