TensorFlow训练报错：索引越界，数据集莫名被放大3倍？

Hey there, let’s break down what’s likely causing your dataset to triple in size and trigger index errors, plus actionable fixes to resolve it. This is a super common pitfall when adapting existing code (like Sentdex’s) for custom CNN training pipelines.

Key Areas to Investigate #

1. Check Your Custom Dataset Class Implementation #

The most probable culprit is a misstep in your Dataset subclass:

• Incorrect __len__ method: If you accidentally return len(your_data) * 3 instead of just len(your_data), this will report a 3x larger dataset size to the DataLoader. Double-check this method—even a tiny typo here can throw everything off.
• Buggy __getitem__ method: If you’re returning multiple samples (e.g., three augmented versions) instead of a single sample per index, the DataLoader will treat each of those as separate entries, inflating the total dataset size. For example, if your __getitem__ looks like return (img1, lbl1), (img2, lbl2), (img3, lbl3), this will split each index into three separate data points.

2. Look for Accidental Data Duplication #

• Repeated data loading: Have you accidentally loaded the same numpy array dataset three times? For example, if you have code like dataset = np.load('data.npy') + np.load('data.npy') + np.load('data.npy') or keep appending the dataset to itself in a loop, this will directly triple your sample count.
• Duplicate dataset initialization: If you’re creating multiple instances of your Dataset class and combining them (e.g., train_data = MyDataset() + MyDataset() + MyDataset()), that’s an obvious cause of the 3x size increase.

3. Debug the DataLoader & Training Loop #

• Hardcoded dataset length: If your training/validation loop uses a hardcoded value like range(1000) instead of dynamically using len(dataloader) or len(train_dataset), it’ll clash with the inflated 3000-sample dataset. Conversely, if your Dataset reports 1000 samples but actually has 3000 under the hood, accessing index 1000+ will trigger an out-of-bounds error.
• DataLoader parameter quirks: While rare, incorrect num_workers settings (especially with shared memory for numpy arrays) can sometimes cause unexpected data duplication. Test with num_workers=0 first to rule out multi-process loading issues.

Quick Debugging Tips #

• Add print statements: Insert print(len(your_data)) right after loading your numpy dataset, print(self.__len__()) in your Dataset class, and print(len(train_dataloader)) before starting training. This will help you trace exactly where the size triples.
• Test with a tiny dataset: Reduce your dataset to 10 samples and run your pipeline—if it becomes 30 samples, you’ll narrow down the problematic code block much faster.
• Inspect sample indices: In your __getitem__ method, print the idx parameter and the corresponding data you’re returning. This can reveal if indices are being reused or mapped incorrectly.

Start with the Dataset class checks first—those are the most frequent sources of this kind of issue. Once you pinpoint where the size is inflating, fixing the index error will follow naturally.

内容的提问来源于stack exchange，提问作者KatharsisHerbie