训练自搭建Inception V3时出现Graph execution error如何解决？

问题背景 #

• 从零搭建Inception V3模型，基于Kaggle平台公开的微软猫狗分类数据集开展训练时，抛出Graph execution error报错
• 初步判断报错与数据集相关，疑似存在非RGB格式图像，不排除其他触发因素

相关代码片段 #

模型编译代码 #

model.compile(optimizer=Adam(learning_rate=0.0001),loss = 'categorical_crossentropy', metrics= ['accuracy'])

数据预处理代码 #

train_batches = ImageDataGenerator(preprocessing_function=tf.keras.applications.inception_v3.preprocess_input) \
    .flow_from_directory(directory=trin_path, target_size=(299,299), classes=['dogs', 'cats'], batch_size=10)
valid_batches = ImageDataGenerator(preprocessing_function=tf.keras.applications.inception_v3.preprocess_input) \
    .flow_from_directory(directory=valid_path, target_size=(299,299), classes=['dogs', 'cats'], batch_size=10)
test_batches = ImageDataGenerator(preprocessing_function=tf.keras.applications.inception_v3.preprocess_input) \
     .flow_from_directory(directory=test_path, target_size=(299,299), classes=['dogs', 'cats'], batch_size=10, shuffle=False)

模型训练代码 #

r = model.fit(x=train_batches, validation_data=valid_batches, epochs=5)

完整报错信息 #

Epoch 1/5
 72/500 [===>..........................] - ETA: 1:14 - loss: 0.6929 - accuracy: 0.6042/usr/local/lib/python3.7/dist-packages/PIL/TiffImagePlugin.py:770: UserWarning: Possibly corrupt EXIF data.  Expecting to read 32 bytes but only got 0. Skipping tag 270
/usr/local/lib/python3.7/dist-packages/PIL/TiffImagePlugin.py:770: UserWarning: Possibly corrupt EXIF data.  Expecting to read 5 bytes but only got 0. Skipping tag 271
/usr/local/lib/python3.7/dist-packages/PIL/TiffImagePlugin.py:770: UserWarning: Possibly corrupt EXIF data.  Expecting to read 8 bytes but only got 0. Skipping tag 272
/usr/local/lib/python3.7/dist-packages/PIL/TiffImagePlugin.py:770: UserWarning: Possibly corrupt EXIF data.  Expecting to read 8 bytes but only got 0. Skipping tag 282
/usr/local/lib/python3.7/dist-packages/PIL/TiffImagePlugin.py:770: UserWarning: Possibly corrupt EXIF data.  Expecting to read 8 bytes but only got 0. Skipping tag 283
/usr/local/lib/python3.7/dist-packages/PIL/TiffImagePlugin.py:770: UserWarning: Possibly corrupt EXIF data.  Expecting to read 20 bytes but only got 0. Skipping tag 306
/usr/local/lib/python3.7/dist-packages/PIL/TiffImagePlugin.py:770: UserWarning: Possibly corrupt EXIF data.  Expecting to read 48 bytes but only got 0. Skipping tag 532
/usr/local/lib/python3.7/dist-packages/PIL/TiffImagePlugin.py:788: UserWarning: Corrupt EXIF data.  Expecting to read 2 bytes but only got 0. 
  warnings.warn(str(msg))
500/500 [==============================] - ETA: 0s - loss: 0.6609 - accuracy: 0.6318
---------------------------------------------------------------------------
UnknownError                              Traceback (most recent call last)
<ipython-input-90-bd0e48768399> in <module>()
----> 1 r = model.fit(x=train_batches,validation_data=valid_batches,epochs=5)

UnknownError: Graph execution error:

2 root error(s) found.
  (0) UNKNOWN:  UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x7fb8672df290>
PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x7fb8672df290>
     [[{{node PyFunc}}]]
     [[IteratorGetNext]]
  (1) CANCELLED:  Function was cancelled before it was started
0 successful operations.
0 derived errors ignored. [Op:__inference_test_function_69081]

根因分析与解决方法 #

报错栈明确抛出PIL.UnidentifiedImageError，属于典型的数据集加载错误，和模型结构、编译参数无关，和前期的数据集问题猜测吻合。

• 核心原因：所用的微软猫狗数据集本身存在若干无效文件，包括零字节损坏图像、EXIF信息损坏的图片、CMYK/灰度等非RGB格式图像，甚至混入了非图像格式的文件。ImageDataGenerator.flow_from_directory默认不会提前校验文件有效性，迭代到坏文件时会直接终止训练，训练前期弹出的EXIF损坏警告就是明确的前兆。报错信息里第二个CANCELLED类型错误是首个错误触发后框架自动终止后续计算产生的连带报错，不需要单独处理。
  解决方法：训练前先执行全数据集清洗，遍历所有文件，能正常打开的统一转换为RGB格式，无法打开的损坏文件直接删除，参考代码如下：

  import os
  from PIL import Image
  
  def clean_dataset(target_dir):
      invalid_files = []
      for root, _, files in os.walk(target_dir):
          for f in files:
              f_path = os.path.join(root, f)
              try:
                  with Image.open(f_path) as img:
                      # 强制转RGB，统一通道格式
                      rgb_img = img.convert("RGB")
                      rgb_img.save(f_path)
              except Exception as e:
                  invalid_files.append(f_path)
                  print(f"发现无效文件：{f_path}，错误信息：{str(e)}")
      # 移除所有无效文件
      for f in invalid_files:
          os.remove(f)
      print(f"数据集清洗完成，共移除{len(invalid_files)}个无效文件")
  
  # 分别清洗训练、验证、测试集目录
  clean_dataset(train_path)
  clean_dataset(valid_path)
  clean_dataset(test_path)
  

• 次要代码问题：预处理代码中训练集路径参数写为trin_path，属于拼写错误，需要和实际定义的训练集路径变量（通常为train_path）保持一致，避免路径指向错误目录加载到无关文件。
• 可选优化方案：如果不想提前清洗数据集，可以替换为TensorFlow官方的image_dataset_from_directory接口做数据加载，支持指定通道格式，高版本TF还支持自动跳过损坏文件，配置参考：

  import tensorflow as tf
  # 加载数据集
  train_batches = tf.keras.utils.image_dataset_from_directory(
      train_path,
      image_size=(299, 299),
      batch_size=10,
      color_mode="rgb",
      label_mode="categorical",
      class_names=['dogs', 'cats']
  )
  valid_batches = tf.keras.utils.image_dataset_from_directory(
      valid_path,
      image_size=(299, 299),
      batch_size=10,
      color_mode="rgb",
      label_mode="categorical",
      class_names=['dogs', 'cats']
  )
  # 映射InceptionV3预处理逻辑
  def preprocess(image, label):
      return tf.keras.applications.inception_v3.preprocess_input(image), label
  train_batches = train_batches.map(preprocess)
  valid_batches = valid_batches.map(preprocess)
  

内容的提问来源于stack exchange，提问作者Md Mahadi Hasan Sany