SSD-MobileNet-V1模型重训配置疑问及训练优化咨询

Hey there! Let's break down your questions about tuning the ssd_mobilenet_v1_0.75_depth_coco model for single-class detection, ARM CPU deployment, and fixing that stubborn loss issue. First, recap your setup: you're training a single-class detector with 300x300 input images, and after 40k steps, loss is stuck at 5.

#1. What Can Be Modified in the pipeline.config? #

Nearly every section of the config is configurable, with many tweaks directly supporting your single-class, ARM-focused goals. Here are the most impactful areas:

• Model Core:
  • num_classes: You already set this to 1 (perfect for your use case).
  • anchor_generator: Adjust aspect_ratios to match only your target object's shape (e.g., remove 2.0/0.5 if your object is square) to cut unnecessary anchor calculations—great for boosting ARM inference speed. Tweak min_scale/max_scale if your target is consistently small or large.
  • post_processing: Tune NMS thresholds (score_threshold, iou_threshold) to reduce false positives and speed up inference.
• Loss Function:
  • hard_example_miner: Adjust parameters like num_hard_examples or max_negatives_per_positive to balance training stability and compute load (critical for CPU training).
• Training Config:
  • batch_size, num_steps: We’ll dive into these in question 3.
  • optimizer: Learning rate, decay schedule, and optimizer type can make a huge difference in loss convergence.
  • data_augmentation_options: Add/remove augmentations tailored to your dataset (e.g., random_brightness if your object varies in lighting, but avoid over-augmenting small datasets).
• Feature Extractor: As we’ll discuss next, this section isn’t locked—you can tweak it for ARM efficiency.

#2. Can We Modify Feature Extractor Parameters? #

No, you’re not limited to only the classification layer—it all depends on your config settings:

• Looking at your config, you have override_base_feature_extractor_hyperparams: true and load_all_detection_checkpoint_vars: false. This means you can modify feature extractor parameters (like depth_multiplier, conv_hyperparams) while fine-tuning from the pre-trained checkpoint.
• Your current depth_multiplier: 0.75 is already a lightweight setting ideal for ARM CPUs. If you need even more speed, you could drop it to 0.5 (expect a small accuracy tradeoff).
• If you want to freeze the feature extractor entirely (to speed up training and reduce compute), add this line to train_config:

  freeze_variables: ["FeatureExtractor/*"]
  

  This locks the base MobileNet layers, so only the detection heads (classification/regression) are trained.

#3. Critical Training Parameters on 16GB CPU + Batch Size/Num Steps #

For CPU training with 16GB RAM, these parameters are make-or-break:

• Data Loading: Increase num_readers in train_input_reader to 4 (from 1) to avoid data bottlenecks—CPUs handle parallel loading better than you might think.
• Batch Size: Your current batch_size: 24 is reasonable, but monitor RAM usage during training. If you hit out-of-memory errors, drop it to 16. For single-class detection, smaller batches can sometimes lead to more stable convergence on CPU.
• Optimizer & Learning Rate: Your initial learning rate of 0.004 might be too high for a single-class task (the pre-trained model was trained on 90 classes). Try lowering it to 0.001, and adjust the decay schedule to decay_steps: 10000 (slower decay) to give the model more time to adapt to your single class.
• Hard Example Mining: num_hard_examples: 3000 is computationally heavy for CPU. Reduce it to 1000 to cut training time without major accuracy loss. You could also lower max_negatives_per_positive to 2 to reduce the ratio of negative examples, which might help bring down that stuck loss.

Recommended Batch Size & Num Steps:

• Batch Size: 16-24 (stick with 24 if your CPU can handle it; drop to 16 if you hit OOM).
• Num Steps: 40k steps isn’t enough if loss is stuck at 5. After adjusting learning rate and hard example mining, train for 60k-80k steps, or stop when loss consistently drops below 1.

Bonus: Fixing Your Stuck Loss at 5 #

Beyond the above tweaks, check these:

• Anchor Matching: If your target object’s size/shape doesn’t match the default anchors, the model can’t learn effectively. Adjust anchor_generator’s aspect_ratios and min_scale/max_scale to match your object.
• Dataset Quality: Double-check that your TFRecords and label map are correctly formatted, and that you have enough positive examples (aim for at least 1k-2k labeled samples for single-class detection).
• Fine-Tuning Scope: Try setting load_all_detection_checkpoint_vars: true to fine-tune the entire model (not just heads)—this can help the feature extractor adapt better to your specific object.

Your Current Config File #

model { ssd { num_classes: 1 box_coder { faster_rcnn_box_coder { y_scale: 10.0 x_scale: 10.0 height_scale: 5.0 width_scale: 5.0 } } matcher { argmax_matcher { matched_threshold: 0.5 unmatched_threshold: 0.5 ignore_thresholds: false negatives_lower_than_unmatched: true force_match_for_each_row: true } } similarity_calculator { iou_similarity { } } anchor_generator { ssd_anchor_generator { num_layers: 6 min_scale: 0.2 max_scale: 0.95 aspect_ratios: 1.0 aspect_ratios: 2.0 aspect_ratios: 0.5 aspect_ratios: 3.0 aspect_ratios: 0.3333 } } image_resizer { fixed_shape_resizer { height: 300 width: 300 } } box_predictor { convolutional_box_predictor { min_depth: 0 max_depth: 0 num_layers_before_predictor: 0 use_dropout: false dropout_keep_probability: 0.8 kernel_size: 1 box_code_size: 4 apply_sigmoid_to_scores: false conv_hyperparams { activation: RELU_6, regularizer { l2_regularizer { weight: 0.00004 } } initializer { truncated_normal_initializer { stddev: 0.03 mean: 0.0 } } batch_norm { train: true, scale: true, center: true, decay: 0.9997, epsilon: 0.001, } } } } feature_extractor { type: "ssd_mobilenet_v1" depth_multiplier: 0.75 min_depth: 16 conv_hyperparams { regularizer { l2_regularizer { weight: 3.99999989895e-05 } } initializer { truncated_normal_initializer { mean: 0.0 stddev: 0.0299999993294 } } activation: RELU_6 batch_norm { decay: 0.97000002861 center: true scale: true epsilon: 0.0010000000475 train: true } } override_base_feature_extractor_hyperparams: true } loss { classification_loss { weighted_sigmoid { } } localization_loss { weighted_smooth_l1 { } } hard_example_miner { num_hard_examples: 3000 iou_threshold: 0.99 loss_type: CLASSIFICATION max_negatives_per_positive: 3 min_negatives_per_image: 0 } classification_weight: 1.0 localization_weight: 1.0 } normalize_loss_by_num_matches: true post_processing { batch_non_max_suppression { score_threshold: 1e-8 iou_threshold: 0.6 max_detections_per_class: 100 max_total_detections: 100 } score_converter: SIGMOID } } } train_config { batch_size: 24 data_augmentation_options { random_horizontal_flip { } } data_augmentation_options { ssd_random_crop { } } optimizer { rms_prop_optimizer: { learning_rate: { exponential_decay_learning_rate { initial_learning_rate: 0.004 decay_steps: 5000 decay_factor: 0.95 } } momentum_optimizer_value: 0.9 decay: 0.9 epsilon: 1.0 } use_moving_average: false } fine_tune_checkpoint: "/content/pretrained_model/model.ckpt" from_detection_checkpoint: true load_all_detection_checkpoint_vars: false num_steps: 40000 } train_input_reader { label_map_path: "/content/classes.pbtxt" tf_record_input_reader { input_path: "/content/gdrive/My Drive/coco_train_300.record" } } eval_config { num_examples: 2693 metrics_set: "coco_detection_metrics" use_moving_averages: false num_visualizations: 20 } eval_input_reader { label_map_path: "/content/classes.pbtxt" shuffle: false num_readers: 1 tf_record_input_reader { input_path: "/content/gdrive/My Drive/coco_val_300.record" } }

内容的提问来源于stack exchange，提问作者tobix10