使用Keras ImageDataGenerator适配3D CNN时的格式错误排查

Hey there! Let's work through this input error you're hitting with your 3D CNN. I've been in your shoes before—using ImageDataGenerator for sequence data can trip up even folks who know their way around Keras, so let's break this down step by step.

核心问题：ImageDataGenerator不支持序列数据 #

You're absolutely right that 3D CNNs expect a 5D tensor with shape (batch_size, num_frames, height, width, channels). But here's the catch: ImageDataGenerator is built for single 2D images, so it outputs a 4D tensor (batch_size, height, width, channels) by default. It can't automatically group your 16 consecutive frames into the sequence shape your model needs—that's why you're seeing input mismatches.

解决方案：自定义序列数据生成器 #

Instead of relying on ImageDataGenerator, we'll build a custom data generator that reads your frame sequences directly. Here's how to do it in Spyder, tailored to your folder structure:

1. 先调整数据集结构（可选但推荐）

To make it easier to grab consecutive frames, organize your dataset so each video's 16 frames live in their own subfolder under the class directory:

your_dataset/
  class_A/
    video_001/
      frame_001.png
      frame_002.png
      ...
      frame_016.png
    video_002/
      ...
  class_B/
    ...

This lets us loop through each video folder and pull exactly the 16 frames we need.

2. 编写自定义Keras Sequence生成器

Keras' Sequence class is perfect for this—it handles batch loading efficiently and works seamlessly with model.fit(). Add this code to your Spyder script:

from keras.utils import Sequence
import numpy as np
import os
from PIL import Image

class VideoSequenceGenerator(Sequence):
    def __init__(self, base_dir, batch_size, target_size=(80,100), num_frames=16, shuffle=True):
        self.base_dir = base_dir
        self.batch_size = batch_size
        self.target_size = target_size
        self.num_frames = num_frames
        self.shuffle = shuffle
        
        # Map class names to numeric labels
        self.class_names = sorted(os.listdir(base_dir))
        self.class_to_label = {name: idx for idx, name in enumerate(self.class_names)}
        
        # Collect all valid video folders (with at least num_frames frames)
        self.video_folders = []
        self.labels = []
        for class_name in self.class_names:
            class_dir = os.path.join(base_dir, class_name)
            for video_folder in os.listdir(class_dir):
                full_video_path = os.path.join(class_dir, video_folder)
                frame_count = len([f for f in os.listdir(full_video_path) if f.endswith(('.png', '.jpg'))])
                if frame_count >= num_frames:
                    self.video_folders.append(full_video_path)
                    self.labels.append(self.class_to_label[class_name])
        
        # Shuffle data on epoch start
        self.on_epoch_end()

    def __len__(self):
        # Number of batches per epoch
        return int(np.ceil(len(self.video_folders) / self.batch_size))

    def __getitem__(self, idx):
        # Grab the current batch of video folders and labels
        batch_folders = self.video_folders[idx*self.batch_size : (idx+1)*self.batch_size]
        batch_labels = self.labels[idx*self.batch_size : (idx+1)*self.batch_size]
        
        # Initialize 5D batch tensor: (batch_size, num_frames, height, width, channels)
        batch_data = np.zeros((len(batch_folders), self.num_frames, *self.target_size, 1), dtype=np.float32)
        
        for i, folder in enumerate(batch_folders):
            # Get sorted frame files to maintain sequence order
            frame_files = sorted([f for f in os.listdir(folder) if f.endswith(('.png', '.jpg'))])
            # Take first 16 frames (you can modify this to pick random consecutive frames later)
            selected_frames = frame_files[:self.num_frames]
            
            for j, frame_file in enumerate(selected_frames):
                frame_path = os.path.join(folder, frame_file)
                # Load frame as grayscale, resize, normalize
                img = Image.open(frame_path).convert('L')
                img = img.resize(self.target_size)
                img_array = np.array(img) / 255.0  # Normalize to 0-1 range
                # Add channel dimension to match (80,100,1)
                batch_data[i, j] = np.expand_dims(img_array, axis=-1)
        
        return batch_data, np.array(batch_labels)

    def on_epoch_end(self):
        if self.shuffle:
            # Shuffle video folders and labels together
            combined = list(zip(self.video_folders, self.labels))
            np.random.shuffle(combined)
            self.video_folders, self.labels = zip(*combined)

3. 初始化生成器并匹配模型输入

Now set up your generator and make sure your model's input layer matches the 5D shape:

# Initialize training generator
train_generator = VideoSequenceGenerator(
    base_dir='path/to/your/train_dataset',
    batch_size=8,  # Adjust based on your GPU memory
    target_size=(80,100),
    num_frames=16,
    shuffle=True
)

# Build your 3D CNN with the correct input shape
from keras.models import Model
from keras.layers import Input, Conv3D, MaxPooling3D, Flatten, Dense

input_layer = Input(shape=(16, 80, 100, 1))  # (num_frames, height, width, channels)
x = Conv3D(32, kernel_size=(3,3,3), activation='relu')(input_layer)
x = MaxPooling3D(pool_size=(2,2,2))(x)
x = Flatten()(x)
x = Dense(64, activation='relu')(x)
output_layer = Dense(len(train_generator.class_names), activation='softmax')(x)

model = Model(inputs=input_layer, outputs=output_layer)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

4. Train your model with the generator

Finally, train using the custom generator—no more input shape mismatches:

model.fit(
    train_generator,
    epochs=15,
    steps_per_epoch=len(train_generator)
)

Quick Tips to Avoid Headaches #

• Consistent Sequence Order: Always sort your frame files (like frame_001.png to frame_016.png) to keep the temporal sequence correct.
• Data Augmentation: If you want to add augmentations (flips, shifts), apply the same transformation to all frames in a sequence—don't randomize per frame, or you'll break the temporal logic.
• GPU Memory: If you get out-of-memory errors, reduce your batch_size (try 4 or 2 instead of 8).

内容的提问来源于stack exchange，提问作者Meme