Great questions about TensorFlow Dataset behavior—these are super common pitfalls when working with tf.data, so let's break them down clearly.

1. How does the order of batch(), shuffle(), repeat() affect output? #

The order of these operations completely changes how your data is processed across epochs and batches. Here's a breakdown of the most common combinations:

• shuffle() -> batch() -> repeat() (Recommended for most training workflows)
  First, we shuffle all individual elements of the original dataset. Then we split the shuffled elements into batches. Finally, we repeat this entire process for each epoch.

  • Each epoch gets a fresh shuffle of the original data, so batch contents and order vary across epochs.
  • No duplicate elements from the original dataset will appear in the same batch (unless your raw data has duplicates), since we shuffle before batching.

• repeat() -> shuffle() -> batch() (Your current workflow)
  We first repeat the original dataset N times (or infinitely if no count is given), creating a long sequence of duplicated data. Then we shuffle this extended sequence, then split into batches.

  • Since we're shuffling a dataset full of repeated elements, it's easy to get duplicate elements in batches or across batches—especially with small datasets.
  • The shuffle operation uses a buffer; if the buffer is smaller than your original dataset size, it will start pulling in repeated elements before the original set is fully shuffled, leading to more duplicates.

• batch() -> shuffle() -> repeat()
  First, split the original dataset into fixed batches (preserving the order of elements within each batch). Then shuffle the order of these batches. Finally, repeat the shuffled batch sequence for each epoch.

  • Batch contents stay the same across epochs—only the order of batches changes.
  • This is useful if you need consistent batch contents but want to vary the order they're processed in, but it's not ideal for training since individual elements aren't shuffled.

2. Why do duplicate elements appear when using repeat() -> shuffle() -> batch() on small datasets? #

Let's unpack exactly what's causing this:

1. Repeat creates duplicated data first
   When you call repeat() first, you're essentially concatenating your small dataset with itself multiple times (or infinitely). For example, if your dataset is [A,B,C], repeat(2) turns it into [A,B,C,A,B,C]. Now your dataset is full of duplicates right from the start.

2. Shuffle's buffer limitation
   The shuffle() operation works by filling an internal buffer with elements, then randomly sampling from that buffer (refilling with new elements as samples are taken). If your buffer size is smaller than your original dataset size (e.g., buffer size 2 for a 3-element dataset), the buffer will quickly include repeated elements. For example, after pulling A and B from the buffer, it will refill with the next element in the repeated sequence—C, then A again. Now the buffer has C and A, so you might sample A again before seeing the original C a second time.

3. Infinite repeat amplifies the issue
   If you're using repeat() without a count (infinite repeat), your dataset never ends. The shuffle buffer will keep pulling in repeated elements forever, so duplicates are guaranteed to pop up regularly.

Fixes to try: #

• Switch to shuffle() -> batch() -> repeat(): This is the standard training workflow. Shuffling the original data first ensures each epoch gets a fresh, unique order of elements, and batching after shuffle prevents duplicate original elements from ending up in the same batch.
• Match shuffle() buffer size to your dataset size: If you must keep your current order, set buffer_size equal to the number of elements in your original dataset. This way, the shuffle buffer contains all unique elements before any duplicates from repeat() are added, reducing the chance of duplicate elements in batches.
• Specify a finite repeat count: Instead of infinite repeat, use repeat(num_epochs) so the dataset ends after a fixed number of epochs. This limits how many times duplicates are introduced.

内容的提问来源于stack exchange，提问作者Miladiouss