Docker部署Keras模型预测精度异常下降问题排查

问题表现 #

• 本地环境运行模型预测，精度稳定可达0.8，本地运行效果示例：
  
• 完全相同的模型结构、权重文件部署到Docker容器后，使用Keras原生接口加载权重时，预测精度从0.8骤降至约0.3，对应运行示例：
  

已开展的排查操作 #

• 验证权重文件完整性：排除拷贝过程损坏的可能，将权重文件压缩后传入容器，内部解压再加载，预测精度仍为0.3
• 验证权重文件有效性：直接在Docker容器内完成模型全流程训练，产出的模型预测精度依然只有0.3
• 更换权重序列化方式测试：改用Pickle序列化保存权重，拷贝到容器后加载预测，精度恢复至约0.78（接近本地水平），对应运行示例：
  

现有环境配置 #

基础版本信息 #

• Docker版本：v20.10.13、v20.10.14

Dockerfile配置 #

FROM continuumio/anaconda3

ENV APP_HOME /modelo_docker
WORKDIR $APP_HOME
COPY . $APP_HOME

RUN pip install --upgrade pip

RUN apt-get update
RUN apt-get install ffmpeg libsm6 libxext6  -y

RUN pip install -r requirements.txt

CMD ["python", "embeddedtnet.py"]

依赖库版本 #

uvicorn==0.17.6
fastapi==0.78.0
tensorflow==2.8.1
tensorflow-text==2.8.1
seaborn==0.11.2
scikit-learn==1.1.1
art==5.6
matplotlib==3.5.2
opencv-python==4.5.5.64
pandas==1.4.2
waitress==2.1.1
tensorflow-addons==0.17.0
pytesseract
nltk==3.7
requests==2.27.1
numpy==1.22.4
contractions==0.1.72
unidecode==1.3.4
protobuf~=3.19.0

核心模型结构 #

class LSTMBLock(tf.keras.Model):
    def __init__(self, vocabulary):
        super(LSTMBLock, self).__init__()

        self.vectorization_layer = tf.keras.layers.experimental.preprocessing.TextVectorization(
            max_tokens=2000, standardize=None, output_mode='int', output_sequence_length=200, name='vectorization_layer')

        self.vectorization_layer.set_vocabulary(vocabulary)

        self.from_ragged_to_dense = tf_text.keras.layers.ToDense(
            pad_value=0, mask=True)

        self.embedding = tf.keras.layers.Embedding(
            input_dim=len(vocabulary), output_dim=200, mask_zero=True)

        self.maxpooling = tf.keras.layers.MaxPooling1D(pool_size=3)

        self.conv1 = tf.keras.layers.Conv1D(
            filters=32, kernel_size=3, padding='same', activation='relu')

        self.spadrop = tf.keras.layers.SpatialDropout1D(0.25)

        self.bidirectionalLSTM = tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64, activation='tanh', recurrent_activation='sigmoid',recurrent_dropout=0, dropout=0.25, kernel_initializer='glorot_uniform', return_sequences=True),
                                                               merge_mode='concat', name="bidirectional")

        self.conv2 = tf.keras.layers.Conv1D(
            filters=32, kernel_size=3, padding='same', activation='relu')

        self.flatten = tf.keras.layers.Flatten()

    def call(self, inputs: List[str]) -> tf.Tensor:

        x = self.vectorization_layer(inputs)
        x = self.from_ragged_to_dense(x)
        x = self.embedding(x)
        x = self.maxpooling(x)
        x = self.conv1(x)
        x = self.spadrop(x)
        x = self.bidirectionalLSTM(x)
        x = self.conv2(x)
        x = self.flatten(x)

        return x

待解决问题 #

1. Keras原生接口加载权重时精度骤降的根本原因
2. Pickle序列化方式加载权重可恢复精度的底层逻辑

内容的提问来源于stack exchange，提问作者Manu