基于Keras的CNN模型训练卡在第1个epoch的问题求助

Hey there, let's figure out why your model's stuck on the first epoch and fix it step by step. I've gone through your code and dataset setup, here are the most likely issues and actionable fixes:

1. 过大的Batch Size是头号嫌疑 #

Your batch_size = 500 is way too big for images of size 300x281. Let's do the math: each image is 3002813 = ~250k pixels, and with float32 precision, that's 1MB per image. A batch of 500 would take up ~500MB just for the input data—before adding model parameters, intermediate activations, and gradients. This will either completely clog your GPU memory (leading to silent hangs) or force TensorFlow to use CPU swap space, making training crawl at a snail's pace.

Fix:
Drop the batch size to a reasonable number, starting with 32 or 64. Adjust based on your GPU's VRAM:

batch_size = 32  # Start small, increase if your GPU can handle it

2. 输入图像尺寸非2的幂次，增加计算复杂度 #

Your input size (300x281) isn't a power of 2, which leads to awkwardly sized feature maps after pooling layers. For example, after the first MaxPooling2D, your height goes from 275 to 137 (275//2), then to 65 after the second pooling—these odd dimensions force the model to do extra computations that slow things down.

Fix:
Resize your images to a standard power-of-2 size like 224x224 (used in ImageNet models) or 256x256. This will make pooling operations cleaner and speed up training:

img_width, img_height = 224, 224
input_shape = (img_width, img_height, 3)

3. 模型计算量过大，轻量化处理 #

Your current model uses 7x7 convolution kernels for both Conv2D layers, which are computationally expensive. A 7x7 kernel does 49x more calculations per pixel than a 3x3 kernel. Combined with the large input size, this creates a huge workload for each batch.

Fix:
Switch to smaller 3x3 kernels (the standard for most CNNs) and consider reducing the number of filters if needed:

model = Sequential()
model.add(Conv2D(32, (3, 3), strides=(1, 1), input_shape=input_shape))  # 3x3 instead of 7x7
model.add(BatchNormalization(axis=3))
model.add(Activation('relu'))
model.add(MaxPooling2D((2, 2)))
model.add(Conv2D(64, (3, 3), strides=(1, 1)))  # 3x3 instead of 7x7
model.add(BatchNormalization(axis=3))
model.add(Activation('relu'))
model.add(MaxPooling2D((2, 2)))

4. 检查数据集是否有损坏的图像 #

A common hidden issue is corrupted or unreadable images in your train or test folders. The ImageDataGenerator might hang indefinitely when trying to load a broken file.

Fix:
Write a quick script to validate all images in your dataset:

import os
from PIL import Image

def check_images(directory):
    for root, dirs, files in os.walk(directory):
        for file in files:
            if file.endswith(('.png', '.jpg', '.jpeg')):
                try:
                    img = Image.open(os.path.join(root, file))
                    img.verify()  # Check if image is valid
                except (IOError, SyntaxError) as e:
                    print(f"Bad image: {os.path.join(root, file)}")
                    # Optionally delete the bad file: os.remove(os.path.join(root, file))

check_images(train_dir)
check_images(test_dir)

Run this to find and remove any corrupted images.

5. 确认GPU是否在正常工作 #

If you're training on CPU, a model this size with 22k samples will take forever. Even 8 hours might not be enough for one epoch. Let's confirm if TensorFlow is using your GPU:

Fix:
Add this code snippet at the top of your notebook:

import tensorflow as tf
print("GPU Available:", tf.test.is_gpu_available())

If it returns False, you need to install the GPU version of TensorFlow (matching your CUDA/CuDNN version) to get proper speedups.

6. 改用model.fit()替代fit_generator() #

In newer TensorFlow/Keras versions, fit_generator() is deprecated, and model.fit() works directly with the generators from flow_from_directory. Using the newer API can resolve stability issues.

Fix:
Replace your fit_generator call with:

model.fit(
    train_data,
    steps_per_epoch=train_samples // batch_size,
    epochs=epochs,
    validation_data=test_data,
    verbose=1,
    validation_steps=test_samples // batch_size
)

额外小建议 #

• Start with a smaller number of epochs (like 5) to test if the model can progress past the first epoch before committing to 30.
• Enable verbose=2 in model.fit() to get more detailed logging, which can help you spot where the hang is happening.

Try these fixes one by one (starting with batch size and input size—those are the biggest culprits!) and your model should start training properly.

内容的提问来源于stack exchange，提问作者johnny_bravo