训练植物病害识别模型遇OUT_OF_RANGE错误，请求排查

问题描述 #

构建含38种病害的植物病害识别模型，每种病害约2000张图片。训练时部分epoch因OUT_OF_RANGE错误中断：第1、3、5、7、9轮训练成功，第2、4、6、8、10轮均报错终止。

训练代码 #

import os
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Input

train_dir = 'dataset/train'
valid_dir = 'dataset/valid'
batch_size = 32

train_datagen = ImageDataGenerator(
    rescale=1./255,
    rotation_range=40,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True,
    fill_mode='nearest'
)

valid_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
    train_dir,
    target_size=(150, 150),
    batch_size=batch_size,
    class_mode='categorical'
)

valid_generator = valid_datagen.flow_from_directory(
    valid_dir,
    target_size=(150, 150),
    batch_size=batch_size,
    class_mode='categorical'
)

model = Sequential([
    Input(shape=(150, 150, 3)),
    Conv2D(32, (3, 3), activation='relu'),
    MaxPooling2D(2, 2),
    Conv2D(64, (3, 3), activation='relu'),
    MaxPooling2D(2, 2),
    Conv2D(128, (3, 3), activation='relu'),
    MaxPooling2D(2, 2),
    Flatten(),
    Dense(512, activation='relu'),
    Dense(38, activation='softmax')  # Adjust output units based on the number of disease classes
])

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

history = model.fit(
    train_generator,
    steps_per_epoch=train_generator.samples // batch_size,
    epochs=10,
    validation_data=valid_generator,
    validation_steps=valid_generator.samples // batch_size
)

model.save('plant_disease_model.h5')

class_indices = train_generator.class_indices
disease_names = list(class_indices.keys())
print("Mapping of Class Indices to Disease Names:", class_indices)

终端报错输出 #

Found 70295 images belonging to 38 classes.
Found 17572 images belonging to 38 classes.
2024-04-23 19:50:32.085744: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Epoch 1/10
\.venv\Lib\site-packages\keras\src\trainers\data_adapters\py_dataset_adapter.py:120: UserWarning: Your `PyDataset` class should call `super().__init__(**kwargs)` in its constructor. `**kwargs` can include `workers`, `use_multiprocessing`, `max_queue_size`. Do not pass these arguments to `fit()`, as they will be ignored.
  self._warn_if_super_not_called()
←[1m2196/2196←[0m ←[32m━━━━━━━━━━━━━━━━━━━━←[0m←[37m←[0m ←[1m905s←[0m 411ms/step - accuracy: 0.4608 - loss: 1.8737 - val_accuracy: 0.7432 - val_loss: 0.8556
Epoch 2/10
←[1m   1/2196←[0m ←[37m━━━━━━━━━━━━━━━━━━━━←[0m ←[1m12:02←[0m 329ms/step - accuracy: 0.6875 - loss: 0.78202024-04-23 20:05:37.996528: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
         [[{{node IteratorGetNext}}]]
C:\Users\Admin\AppData\Local\Programs\Python\Python311\Lib\contextlib.py:155: UserWarning: Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches. You may need to use the `.repeat()` function when building your dataset.
  self.gen.throw(typ, value, traceback)
2024-04-23 20:05:38.068817: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
         [[{{node IteratorGetNext}}]]
←[1m2196/2196←[0m ←[32m━━━━━━━━━━━━━━━━━━━━←[0m←[37m←[0m ←[1m0s←[0m 49us/step - accuracy: 0.6875 - loss: 0.7820 - val_accuracy: 0.7500 - val_loss: 0.2462

问题原因与修复方案 #

核心原因 #

flow_from_directory生成的生成器默认遍历完所有数据后就会停止迭代。手动指定steps_per_epoch后，生成器没有循环重复输出数据的机制，第一轮用完所有数据后，后续轮次无法获取足够的batch，触发OUT_OF_RANGE错误。

修复方案1：移除手动指定的步数参数（最简方案） #

Keras会自动根据生成器的总样本数和batch_size计算每轮所需步数，无需手动设置steps_per_epoch和validation_steps。修改model.fit部分代码：

history = model.fit(
    train_generator,
    epochs=10,
    validation_data=valid_generator
)

修复方案2：将生成器转为可重复的tf.data.Dataset #

如果需要保留手动指定步数的逻辑，可将生成器转换为TensorFlow的Dataset并添加repeat()，确保数据循环迭代：

import tensorflow as tf

# 转换训练生成器为可重复的Dataset
train_dataset = tf.data.Dataset.from_generator(
    lambda: train_generator,
    output_types=(tf.float32, tf.float32),
    output_shapes=([None, 150, 150, 3], [None, 38])
).repeat()

# 转换验证生成器为可重复的Dataset
valid_dataset = tf.data.Dataset.from_generator(
    lambda: valid_generator,
    output_types=(tf.float32, tf.float32),
    output_shapes=([None, 150, 150, 3], [None, 38])
).repeat()

# 训练时指定步数
history = model.fit(
    train_dataset,
    steps_per_epoch=train_generator.samples // batch_size,
    epochs=10,
    validation_data=valid_dataset,
    validation_steps=valid_generator.samples // batch_size
)

额外注意 #

• 生成器默认开启shuffle=True，每轮训练会自动打乱数据顺序，无需额外处理。
• 如果手动计算步数，若要包含所有样本，可使用steps_per_epoch=(train_generator.samples + batch_size -1) // batch_size，避免因整除遗漏少量样本。

内容的提问来源于stack exchange，提问作者dev