Hey there! Since you're new to deep learning projects, let's walk through this issue clearly—you've already nailed the training part, so we just need to fix the test phase mismatch.

First, let's recap your error for clarity:

wrote gt roidb to ./data/cache/ImageNetVID_DET_val_gt_roidb.pkl
Traceback (most recent call last):
  File "experiments/dff_rfcn/dff_rfcn_end2end_train_test.py", line 20, in <module>
    test.main()
  File "experiments/dff_rfcn/../../dff_rfcn/test.py", line 53, in main
    args.vis, args.ignore_cache, args.shuffle, config.TEST.HAS_RPN, config.dataset.proposal, args.thresh, logger=logger, output_path=final_output_path)
  File "experiments/dff_rfcn/../../dff_rfcn/function/test_rcnn.py", line 68, in test_rcnn
    roidbs_seg_lens[gpu_id] += x['frame_seg_len']
KeyError: 'frame_seg_len'

What's Causing This? #

The root issue is that the original project is built for video datasets (ImageNetVID), which uses the frame_seg_len field to track video frame segments. Your custom binary dataset is likely an image-only dataset (no video sequence data), so this field doesn't exist in your dataset's roidb (Region of Interest Database) entries. Here are the specific possible causes:

• Incomplete cache cleanup: Even if you deleted old caches, the test script might still be generating a roidb using the original video dataset logic (notice it's writing ImageNetVID_DET_val_gt_roidb.pkl—that's still referencing the original VID dataset!). This means your custom dataset isn't being properly loaded during testing.

• Unmodified video-specific code: The test logic in test_rcnn.py still expects video-related fields like frame_seg_len that don't apply to your image dataset. The original code was written to handle sequential video frames, which your binary dataset doesn't have.

• Txt file format mismatches: If you modified the input txt filenames without updating how the script parses them, the dataset loader might not generate the correct roidb structure. The original txt files probably included video sequence info, while your custom txt only has image paths/labels—so the loader skips creating frame_seg_len (which isn't needed, but the test code still looks for it).

How to Fix It #

Let's go through actionable steps:

1. Force full cache regeneration & dataset config check

   • Delete all cache files related to the original VID dataset (not just the one you saw). Check ./data/cache/ for any files with ImageNetVID in the name, and delete them.
   • Double-check your config file to ensure the test phase is pointing to your custom dataset, not the original ImageNetVID. Look for config fields like config.dataset.name or config.TEST.DATASET and confirm they're set to your binary dataset's name.

2. Remove video-specific logic from test code

   • Open dff_rfcn/function/test_rcnn.py and find the line throwing the error: roidbs_seg_lens[gpu_id] += x['frame_seg_len']
   • Since your dataset doesn't use video segments, you can either:
     • Comment out or delete this line (and any related code that uses roidbs_seg_lens—search the file for other references to this variable)
     • Add a conditional check to skip this line when using your custom dataset, e.g.:

       if 'frame_seg_len' in x:
           roidbs_seg_lens[gpu_id] += x['frame_seg_len']
       

3. Validate your custom txt file parsing

   • Compare your modified txt file structure with the original project's txt files. Ensure each line includes all the info the loader expects (e.g., image path, class label, bounding box coordinates if using detection).
   • Check the dataset loading script (look for files like dataset.py or imagenet_vid.py in the project) and update it to parse your custom txt format correctly, ensuring it doesn't try to add video-related fields to the roidb.

4. Verify training vs test dataset consistency

   • Make sure the test dataset uses the same processing pipeline as your training dataset. If you modified the training data loader but not the test one, that could cause mismatched roidb structures.

内容的提问来源于stack exchange，提问作者PAPAS