如何使用tensorflow lite的object_detector.EfficientDetLite4Spec检查点继续训练

核心原因 #

TensorFlow Lite Model Maker 的object_detector.create接口默认关闭了检查点自动恢复逻辑，需要手动开启对应参数。

修复方法 #

仅需在调用object_detector.create时新增resume_training=True参数即可，修改后的代码如下：

import numpy as np
import os

from tflite_model_maker.config import ExportFormat
from tflite_model_maker import model_spec
from tflite_model_maker import object_detector

import tensorflow as tf
assert tf.__version__.startswith('2')

tf.get_logger().setLevel('ERROR')
from absl import logging
logging.set_verbosity(logging.ERROR)

train_data, validation_data, test_data = object_detector.DataLoader.from_csv('csv_path')

spec = object_detector.EfficientDetLite4Spec(
    uri='/content/model',
    model_dir='/content/drive/MyDrive/MathSymbolRecognition/CheckPoints/',
    hparams='grad_checkpoint=true,strategy=gpus',
    epochs=50, batch_size=3,
    steps_per_execution=1, moving_average_decay=0,
    var_freeze_expr='(efficientnet|fpn_cells|resample_p6)',
    tflite_max_detections=25, strategy=spec_strategy
)

model = object_detector.create(train_data, model_spec=spec, batch_size=3, 
    train_whole_model=True, validation_data=validation_data,
    resume_training=True)

异常排查 #

如果添加参数后仍无法恢复，按以下步骤检查：

• 确认/content/drive/MyDrive/MathSymbolRecognition/CheckPoints/路径下存在名为checkpoint的索引文件，该文件记录了最新检查点的路径，缺失的话TensorFlow无法识别历史训练产物
• 若checkpoint文件丢失，可手动创建，内容格式如下，将路径替换为你最新的检查点文件名即可：

model_checkpoint_path: "ckpt-25"
all_model_checkpoint_paths: "ckpt-20"
all_model_checkpoint_paths: "ckpt-25"

• 确认训练轮次参数epochs大于你已经训练完成的轮次，否则会直接停止训练

内容的提问来源于stack exchange，提问作者Michael Chen