使用TensorFlow和Keras训练Fashion-MNIST模型遇ValueError问题求助

问题场景 #

训练Fashion-MNIST模型时运行代码触发以下错误：

ValueError: When providing an infinite dataset, you must specify the number of steps to run (if you did not intend to create an infinite dataset, make sure to not call repeat() on the dataset).

原代码 #

import tensorflow as tf
import tensorflow_datasets as tfds
import math
import numpy as np
import matplotlib.pyplot as plt
dataset = tfds.load('fashion_mnist', as_supervised=True)
train = dataset['train'];
test = dataset['test'];

def normalize(image, label):
  image = tf.cast(image, tf.float32);
  image = image/255;
  return image, label


train = train.map(normalize);
test = test.map(normalize);

train = train.cache();
test = test.cache();

model = tf.keras.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28)),
    tf.keras.layers.Dense(500, activation='relu'),
    tf.keras.layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy,
              metrics=['accuracy'])

model.fit(train, epochs=10, batch_size=100);

错误回溯 #

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-57-07d41cbbc0f1> in <module>()
----> 1 model.fit(train, epochs=10, batch_size=100);

1 frames
/usr/local/lib/python3.7/dist-packages/keras/engine/data_adapter.py in _validate_args(self, y, sample_weights, steps)
    759       if size == tf.data.experimental.INFINITE_CARDINALITY and steps is None:
    760         raise ValueError(
--> 761             "When providing an infinite dataset, you must specify "
    762             "the number of steps to run (if you did not intend to "
    763             "create an infinite dataset, make sure to not call "

ValueError: When providing an infinite dataset, you must specify the number of steps to run (if you did not intend to create an infinite dataset, make sure to not call `repeat()` on the dataset).

错误原因 #

1. 用tf.data.Dataset作为model.fit输入时，若在fit中指定batch_size，Keras会自动对数据集执行repeat()操作，导致数据集变成无限循环状态，且无法自动推断每轮训练步数，触发报错。
2. 原代码中loss参数传入的是损失类而非实例，后续运行也会引发异常。

解决方案 #

1. 正确处理数据集 #

直接在数据集上完成打乱、分批次操作，不在model.fit中指定batch_size：

# 训练集：缓存+打乱+分批次（Fashion-MNIST训练集共60000条数据，用全量打乱保证随机性）
train = train.cache().shuffle(60000).batch(100)
# 测试集：缓存+分批次
test = test.cache().batch(100)

2. 修正损失函数实例化 #

将loss参数改为损失类的实例：

model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(),
              metrics=['accuracy'])

3. 调整model.fit调用 #

去掉batch_size参数，传入处理好的数据集：

model.fit(train, epochs=10);

完整修正后代码 #

import tensorflow as tf
import tensorflow_datasets as tfds
import math
import numpy as np
import matplotlib.pyplot as plt

dataset = tfds.load('fashion_mnist', as_supervised=True)
train = dataset['train']
test = dataset['test']

def normalize(image, label):
  image = tf.cast(image, tf.float32)
  image = image/255
  return image, label

train = train.map(normalize)
test = test.map(normalize)

# 添加shuffle和batch处理
train = train.cache().shuffle(60000).batch(100)
test = test.cache().batch(100)

model = tf.keras.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28)),
    tf.keras.layers.Dense(500, activation='relu'),
    tf.keras.layers.Dense(10, activation='softmax')
])

# 修正损失函数为实例
model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(),
              metrics=['accuracy'])

# 去掉batch_size参数
model.fit(train, epochs=10)

内容的提问来源于stack exchange，提问作者Osama Mohammed