tf.keras训练数据集可视化报错，验证集正常求排查

我用tf.keras.utils.image_dataset_from_directory加载PetImages目录下的20791张两类图片，成功创建了训练集（16633张）和验证集（4158张），但用matplotlib.pyplot绘制训练集时出错，验证集绘制完全正常。

数据加载代码 #

image_size = (180, 180)
batch_size = 128

train_ds, val_ds = tf.keras.utils.image_dataset_from_directory(
    "PetImages",
    labels="inferred",
    label_mode="binary",
    validation_split=0.2,
    subset="both",
    seed=1337,
    image_size=image_size,
    batch_size=batch_size,
)

运行提示 #

Found 20791 files belonging to 2 classes.
Using 16633 files for training.
Using 4158 files for validation.

可视化代码 #

import matplotlib.pyplot as plt
plt.figure(figsize=(10, 10))

for image, label in train_ds.take(1):
    for i in range(9):
        ax = plt.subplot(3, 3, i + 1)
        plt.imshow(image[i].numpy().astype("uint8"))
        plt.title(int(label[i]))
        plt.axis("off")

报错信息 #

InvalidArgumentError                      Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_12336\4181458072.py in <module>
      5 plt.figure(figsize=(10, 10))
      6 
----> 7 for image, label in train_ds.take(1):
      8     for i in range(9):
      9         ax = plt.subplot(3, 3, i + 1)

~\Anaconda3\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py in __next__(self)
    764   def __next__(self):
    765     try:
---> 766       return self._next_internal()
    767     except errors.OutOfRangeError:
    768       raise StopIteration

~\Anaconda3\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py in _next_internal(self)
    747     # to communicate that there is no more data to iterate over.
    748     with context.execution_mode(context.SYNC):
---> 749       ret = gen_dataset_ops.iterator_get_next(
    750           self._iterator_resource,
    751           output_types=self._flat_output_types,

~\Anaconda3\lib\site-packages\tensorflow\python\ops\gen_dataset_ops.py in iterator_get_next(iterator, output_types, output_shapes, name)
   3014       return _result
   3015     except _core._NotOkStatusException as e:
-> 3016       _ops.raise_from_not_ok_status(e, name)
   3017     except _core._FallbackException:
   3018       pass

~\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py in raise_from_not_ok_status(e, name)
   7207 def raise_from_not_ok_status(e, name):
   7208   e.message += (" name: " + name if name is not None else "")
-> 7209   raise core._status_to_exception(e) from None  # pylint: disable=protected-access
   7210 
   7211 

InvalidArgumentError: {{function_node __wrapped__IteratorGetNext_output_types_2_device_/job:localhost/replica:0/task:0/device:CPU:0}} Input is empty.
     [[{{node decode_image/DecodeImage}}]] [Op:IteratorGetNext]

<Figure size 1000x1000 with 0 Axes>

错误原因与解决方法 #

核心原因 #

报错里的Input is empty明确指向：训练集划分到的图片文件中存在空文件或损坏无法解码的图片，而验证集刚好没包含这类文件，所以能正常运行。

解决步骤 #

1. 定位并清理坏文件

   • 遍历PetImages目录下的所有文件，检查文件大小，直接删除大小为0的文件
   • 也可以用PIL库尝试打开图片，捕获异常来找出损坏文件：

     import os
     from PIL import Image
     
     root_dir = "PetImages"
     for subdir, _, files in os.walk(root_dir):
         for file in files:
             file_path = os.path.join(subdir, file)
             try:
                 with Image.open(file_path) as img:
                     img.verify()
             except (IOError, SyntaxError):
                 print(f"删除损坏文件: {file_path}")
                 os.remove(file_path)
     

2. 重新生成数据集
   清理完成后，重新运行数据加载代码，再执行可视化逻辑即可。

3. 可选：添加数据集过滤逻辑
   如果不想手动清理，可以在数据集加载后添加过滤步骤，跳过无法解码的图片：

   def is_valid_image(image, label):
       # 检查图片是否为空
       return tf.reduce_all(tf.not_equal(image, 0))
   
   train_ds = train_ds.filter(is_valid_image)
   

内容的提问来源于stack exchange，提问作者Balaji M Srinivasan