使用image_dataset_from_directory加载数据集时遇UnicodeDecodeError

问题场景 #

加载自定义图像数据集时持续触发UnicodeDecodeError，调整代码参数、修改目录结构均无法解决该问题。

尝试过的目录结构 #

第一种：

main_dir 
--train
----class1
----class2
--val
--test

第二种：

main_dir
...class1
...class2
...

使用的代码 #

import tensorflow as tf
from tensorflow import keras
from keras.utils.image_dataset import image_dataset_from_directory

in_folder = 118
size = 256
data_dir = r"C:\programming\Python\Programs\fonts_render\AI_train"

train_dataset = image_dataset_from_directory(
    data_dir + r"\train",
    labels="inferred",
    label_mode='categorical',
    seed=2,
    image_size=(size, size),
    batch_size=in_folder
)

val_ds = tf.keras.utils.image_dataset_from_directory(
    data_dir + r"\val",
    labels="inferred",
    label_mode='categorical',
    seed=2,
    image_size=(size, size),
    batch_size=in_folder)

报错信息 #

Traceback (most recent call last):
  File "c:\programming\Python\Programs\fonts_render\AI.py", line 12, in <module>
    train_dataset = image_dataset_from_directory(
  File "C:\Python310\lib\site-packages\keras\utils\image_dataset.py", line 210, in image_dataset_from_directory
    image_paths, labels, class_names = dataset_utils.index_directory(
  File "C:\Python310\lib\site-packages\keras\utils\dataset_utils.py", line 573, in index_directory
    partial_filenames, partial_labels = res.get()
  File "C:\Python310\lib\multiprocessing\pool.py", line 771, in get
    raise self._value
  File "C:\Python310\lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "C:\Python310\lib\site-packages\keras\utils\dataset_utils.py", line 644, in index_subdirectory
    for root, fname in valid_files:
  File "C:\Python310\lib\site-packages\keras\utils\dataset_utils.py", line 619, in iter_valid_files
    for root, _, files in sorted(walk, key=lambda x: x[0]):
  File "C:\Python310\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 876, in walk_v2
    if is_directory(full_path):
  File "C:\Python310\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 689, in is_directory
    return is_directory_v2(dirname)
  File "C:\Python310\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 703, in is_directory_v2
    return _pywrap_file_io.IsDirectory(compat.path_to_bytes(path))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8 in position 76: invalid start byte

问题原因与解决方法 #

这个错误和目录结构无关，核心原因是路径中存在无法用UTF-8解码的字符——比如中文、全角符号、生僻字或编码不规范的文件名/目录名，报错里的byte 0xa8大概率对应GBK编码的特殊字符。

解决步骤：

• 排查data_dir路径下所有子目录、图像文件名，替换掉所有非ASCII字符（中文、全角标点、特殊符号等），改用英文、数字和半角符号。
• 路径中的空格建议用下划线替代，避免潜在的解析问题。
• 拼接路径时推荐用os.path.join()替代字符串拼接，减少路径分隔符错误：

  import os
  train_dir = os.path.join(data_dir, "train")
  val_dir = os.path.join(data_dir, "val")
  

内容的提问来源于stack exchange，提问作者nosokvkokose