调用TFServing API触发std::bad_alloc致Docker容器崩溃

调用TFServing预测API时，Docker容器立即崩溃，错误日志如下：

2022-10-05 08:22:19.091237: I tensorflow_serving/model_servers/server.cc:442] Exporting HTTP/REST API at:localhost:8601 ...
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

环境信息：

• TFServing部署在Docker容器中
• 请求来自Flask服务
• 宿主机虚拟机配备16GB内存

相关代码及配置文件：

server.py #

from flask import current_app, flash, jsonify, make_response, redirect, request, url_for
from keras_preprocessing.sequence import pad_sequences
from keras_preprocessing.text import Tokenizer
from dotenv import load_dotenv
from loguru import logger
from pathlib import Path
from flask import Flask
import tensorflow as tf
import numpy as np
import requests
import string
import pickle5 as pickle
import nltk
import re
import os



app = Flask(__name__)
load_dotenv()


@app.route("/test")
def index():

    txt = "This is a text"
    output = get_prediction_probability(txt)
    return output

def text_wragling(text):
    x = text.lower()
    x = remove_URL(x)
    x = remove_punct(x)
    x = remove_stopwords(x)
    with open('tokenizer.pickle', 'rb') as handle:
        tokenizer = pickle.load(handle)
    x = tokenizer.texts_to_sequences([x])
    # pad
    x = pad_sequences(x, maxlen=int(os.getenv('NLP__MAXLEN')))
    return x

def remove_URL(text):
    url = re.compile(r"https?://\S+|www.\.S+")
    return url.sub(r"",text)

def remove_punct(text):
    translator = str.maketrans("", "", string.punctuation)
    return text.translate(translator)

def remove_stopwords(text):

    # nltk.download()
    nltk.download('stopwords')
    from nltk.corpus import stopwords
    stop = set(stopwords.words("english"))
    filtered_words = [word.lower() for word in text.split() if word.lower() not in stop]
    return " ".join(filtered_words)


def get_prediction_probability(txt):
    x = text_wragling(txt)
    logger.info("Txt wragling")
    data = {
        "instances": [
            x.tolist()
        ]
    }
    #logger.info(data)
    logger.info("Get prediction from model")
    response = requests.post("http://localhost:8601/v1/models/nlp_model/labels/production:predict", json=data)
    probability = (np.asarray(response.json()['predictions']).max(axis=1))
    pred = np.asarray(response.json()['predictions']).argmax(axis=1)
    with open('labelenconder.pickle', 'rb') as handle:
        le = pickle.load(handle)
    pred = le.classes_[pred]
    prediction = pred[0]
    return {
        "prediction": prediction,
        "probability": probability[0]
    }


if __name__ == '__main__':
    #test()
   app.run(host='0.0.0.0')

Dockerfile #

FROM tensorflow/serving
EXPOSE 8601

docker-compose.yml #

version: '3'

services:
  tfserving:
    container_name: tfserving
    build: ..
    ports:
      - "8601:8601"
    volumes:
      - ./model.config:/models/model.config
      - ../model:/models/model
    environment:
      - TENSORFLOW_SERVING_REST_API_PORT=8061
      - TENSORFLOW_SERVING_MODEL_NAME=model

      - TENSORFLOW_MODEL_BASE_PATH=/models/model/
    entrypoint: [ "bash", "-c", "tensorflow_model_server --rest_api_port=8601  --allow_version_labels_for_unavailable_models --model_config_file=/models/model.config"]

model.config #

model_config_list {
  config {
    name: 'nlp_model'
    base_path: '/models/model/'
    model_platform: 'tensorflow'
    model_version_policy {
      specific {
        versions: 1
        versions: 2
      }
    }
    version_labels {
      key: 'production'
      value: 1
    }
    version_labels {
      key: 'beta'
      value: 2
    }
  }
}

std::bad_alloc本质是内存分配失败，结合你的配置和代码，主要排查以下几个方向：

1. 请求数据格式错误导致内存溢出 #

你的Flask代码中构造请求数据时，x是经过pad_sequences处理后的二维数组（形状为(1, MAXLEN)），但你在data["instances"]里嵌套了两层列表，导致发送给TFServing的数据结构变成[[[...]]]（三维数组），而模型预期的输入是二维数组（批量的序列数据）。TFServing尝试解析这种不符合预期的输入时，可能错误分配大量内存，最终触发std::bad_alloc。

解决方法：
直接将x.tolist()放入instances列表，无需额外嵌套：

data = {
    "instances": x.tolist()
}

2. Docker容器内存限制未配置 #

虽然宿主机有16GB内存，但如果系统内存紧张或其他进程占用过高，容器可能无法获取足够内存；也可能是系统默认对容器有内存限制。

排查与解决：

• 执行docker stats tfserving查看容器内存使用情况
• 在docker-compose.yml中显式设置容器内存配额，比如分配8GB：

services:
  tfserving:
    # 其他配置不变
    deploy:
      resources:
        limits:
          memory: 8G
        reservations:
          memory: 4G

3. 多模型加载导致内存不足 #

你的model.config中同时加载了版本1和版本2两个模型，如果两个模型体积都很大，会导致容器启动后内存占用过高，接收请求时进一步耗尽内存。

排查与解决：

• 先尝试只加载一个模型版本，修改model.config中的model_version_policy：

model_version_policy {
  specific {
    versions: 1
  }
}

• 检查模型文件大小，确认单个模型是否超出容器内存承载能力。

4. 环境变量与启动参数冲突 #

你的docker-compose中设置了环境变量TENSORFLOW_SERVING_REST_API_PORT=8061，但启动命令中又指定了--rest_api_port=8601，这种冲突可能导致内部配置异常，间接引发问题。

解决方法：
删除冲突的环境变量，保持启动参数一致：

environment:
  # 移除以下两行
  # - TENSORFLOW_SERVING_REST_API_PORT=8061
  # - TENSORFLOW_SERVING_MODEL_NAME=model
  - TENSORFLOW_MODEL_BASE_PATH=/models/model/

内容的提问来源于stack exchange，提问作者Test