神经网络训练停滞：首个Epoch未完成求助

Hey there! I see your training is getting stuck before finishing the first Epoch—super frustrating, right? Let's walk through some common issues and fixes based on your code and the provided screenshot:

Common Causes & Fixes #

1. Data Loading Issues (Most Likely Culprit) #

• Verify Directory Structure: Make sure your train and validation folders follow the required structure for flow_from_directory: each category needs its own subfolder (e.g., /data/train/cat/ and /data/train/dog/ for a binary classification task). If the structure is wrong, the generator might hang trying to find valid data.
• Check for Corrupted Images: Damaged or unreadable images can cause the data generator to freeze. Run this quick script to scan your dataset:

  import os
  from PIL import Image
  
  def validate_images(directory):
      for root, _, files in os.walk(directory):
          for file in files:
              if file.lower().endswith(('.png', '.jpg', '.jpeg')):
                  img_path = os.path.join(root, file)
                  try:
                      with Image.open(img_path) as img:
                          img.verify()  # Check if image is valid
                  except (IOError, SyntaxError) as e:
                      print(f"Corrupted image found: {img_path}")
                      # Optional: Delete or move the bad file
                      # os.remove(img_path)
  
  validate_images('/data/train')
  validate_images('/data/validation')
  

• Replace Deprecated Parameters: samples_per_epoch and nb_val_samples are outdated in newer Keras/TensorFlow versions. Use steps_per_epoch and validation_steps instead, calculated as:

  steps_per_epoch = nb_train_samples // batch_size
  validation_steps = nb_validation_samples // batch_size
  

  This avoids issues where your total sample count isn't perfectly divisible by the batch size.

2. Model & Weight Loading Problems #

• Check Top Model Weight Compatibility: Ensure the weights in /data/fc_model.h5 match exactly with your defined top_model structure. If you trained the top model with a different input shape or layer configuration, loading mismatched weights can cause silent failures. Try commenting out the top_model.load_weights(...) line first—if training runs, the issue is with the pre-trained weights.
• Validate Model Structure: Print your full model summary to confirm all layers are connected correctly:

  print(model.summary())
  

  Look for any unexpected output shapes or disconnected layers that might break the computation graph.

3. Resource Constraints #

• Reduce Batch Size: A batch size of 16 might be too large for your GPU memory. Try dropping it to 8 or 4—this reduces memory usage and can resolve freezing caused by out-of-memory errors (even if you don't see an explicit error message).
• Check CPU/GPU Usage: If you're training on CPU, the first Epoch can take very long (especially with VGG16). Monitor your CPU usage—if it's consistently high, the model is just training slowly, not stuck. If usage is low, then something is blocking the process.

4. Version Compatibility #

• Update to Modern Keras Syntax: fit_generator is deprecated; switch to model.fit() instead (it works with generators too). This avoids potential bugs in older deprecated methods.

Modified Code Example #

Here's your code with the key fixes applied:

from keras import applications
from keras.models import Sequential, Model
from keras.layers import Flatten, Dense, Dropout
from keras.optimizers import SGD
from keras.preprocessing.image import ImageDataGenerator

top_model_weights_path = '/data/fc_model.h5'
img_width, img_height = 150, 150

train_data_dir = '/data/train'
validation_data_dir = '/data/validation'
nb_train_samples = 2000
nb_validation_samples = 800
epochs = 50
batch_size = 8  # Reduced batch size for testing

# Load pre-trained VGG16 base
model = applications.VGG16(weights='imagenet', include_top=False, input_shape=(img_width, img_height, 3))
print('Model loaded.')

# Define top classification model
top_model = Sequential()
top_model.add(Flatten(input_shape=model.output_shape[1:]))
top_model.add(Dense(256, activation='relu'))
top_model.add(Dropout(0.5))
top_model.add(Dense(1, activation='sigmoid'))

# Uncomment this line only after confirming the model runs without weights
# top_model.load_weights(top_model_weights_path)

# Combine base and top model
model = Model(inputs=model.input, outputs=top_model(model.output))

# Freeze first 25 layers
for layer in model.layers[:25]:
    layer.trainable = False

# Compile model
model.compile(loss='binary_crossentropy',
              optimizer=SGD(lr=1e-4, momentum=0.9),
              metrics=['accuracy'])

# Data generators
train_datagen = ImageDataGenerator(
    rescale=1. / 255,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1. / 255)

train_generator = train_datagen.flow_from_directory(
    train_data_dir,
    target_size=(img_height, img_width),
    batch_size=batch_size,
    class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
    validation_data_dir,
    target_size=(img_height, img_width),
    batch_size=batch_size,
    class_mode='binary')

# Calculate training steps
steps_per_epoch = nb_train_samples // batch_size
validation_steps = nb_validation_samples // batch_size

# Train with modern fit() method
model.fit(
    train_generator,
    steps_per_epoch=steps_per_epoch,
    epochs=epochs,
    validation_data=validation_generator,
    validation_steps=validation_steps)

Start with the simplest checks first—verify your data structure and scan for bad images. Then test the model without loading the top weights to rule out weight-related issues. Good luck!

内容的提问来源于stack exchange，提问作者Oussama