如何解决自定义tf.Keras生成器训练CNN时的ValueError:0 is not in range

问题背景 #

对语音样本做实时数据增强时，自定义的语音增强生成器单独测试运行正常，但用于CNN模型训练时出现KeyError: 0错误。

错误栈信息 #

UnknownError                              Traceback (most recent call last)
Cell In [31], line 1
----> 1 history = model.fit(train_aug_gen,
      2           #steps_per_epoch=5,
      3           #validation_steps=5,
      4           validation_data = val_aug_gen,
      5           epochs=num_epochs,verbose=2,          
      6           callbacks = [earlystopping, scheduler, checkpointer])

File ~/miniforge3/envs/env_tf/lib/python3.9/site-packages/keras/utils/traceback_utils.py:70, in filter_traceback.<locals>.error_handler(*args, **kwargs)
     67     filtered_tb = _process_traceback_frames(e.__traceback__)
     68     # To get the full stack trace, call:
     69     # `tf.debugging.disable_traceback_filtering()`
---> 70     raise e.with_traceback(filtered_tb) from None
     71 finally:
     72     del filtered_tb

File ~/miniforge3/envs/env_tf/lib/python3.9/site-packages/tensorflow/python/eager/execute.py:54, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     52 try:
     53   ctx.ensure_initialized()
---> 54   tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
     55                                       inputs, attrs, num_outputs)
     56 except core._NotOkStatusException as e:
     57   if name is not None:

UnknownError: Graph execution error:

2 root error(s) found.
  (0) UNKNOWN:  KeyError: 0
Traceback (most recent call last):

  File "/Users/local/miniforge3/envs/env_tf/lib/python3.9/site-packages/pandas/core/indexes/range.py", line 391, in get_loc
    return self._range.index(new_key)

ValueError: 0 is not in range

自定义生成器核心代码 #

class CustomVoiceAugmentation(tf.keras.utils.Sequence):
    def __init__(self, dir_data, data_df, aug_level, num_classes, shuffle):
        .......
        .......
        print(f"Found {self.data_frame.shape[0]} voices belonging to {self.num_classes} classes") 
       
    def __len__(self):
        return (self.data_len // self.batch_size)

    def __voice_augmentation(self,signalvoice):
            .....
            .....
            .....
            return augmented_voice

    def __get_voice(self,index, batch_x, batch_y, dir_data):
        X = []
        Y = []
        for i, name in enumerate(batch_x):
            label = batch_y[i]
            pathvoice = os.path.join(self.dir_data + '/'+ label+ '/'+ name)
            .....
            .....
            .....
        
        return X, Y
        
    def __to_melspectrogram (self,augmented_voice_signal):
        ......     
        return image

    def __getitem__(self, index):
        
        batch_x = self.data_frame["VoiceName"][index * self.batch_size:(index + 1) * self.batch_size]
        batch_y = self.data_frame["label"][index * self.batch_size:(index + 1) * self.batch_size]
        X ,Y = self.__get_voice(index, batch_x, batch_y, self.dir_data)
        ......
        ......
        ......
        return tf.convert_to_tensor(image), new_Y
    
    
    def on_epoch_end(self):
        if self.shuffle == True:
            self.data_frame = self.data_frame.sample(frac=1).reset_index(drop=True)

问题排查与解决思路 #

这个错误本质是pandas访问DataFrame索引时找不到对应值，结合生成器逻辑，可从以下方向修复：

1. 修正__len__的batch数量计算 #

当前__len__用self.data_len // self.batch_size，若总样本数无法被batch_size整除，最后一批样本会被丢弃，但Keras仍可能尝试访问超出范围的索引。

• 修复代码：

  def __len__(self):
      return (self.data_len + self.batch_size - 1) // self.batch_size
  

  确保所有样本都被覆盖，避免索引越界。

2. 强制DataFrame索引连续 #

初始化或洗牌后，DataFrame的索引可能出现非连续情况，导致切片访问出错。

• 修复：在__init__末尾添加索引重置：

  def __init__(self, dir_data, data_df, aug_level, num_classes, shuffle):
      .......
      self.data_frame = data_df.reset_index(drop=True)
      print(f"Found {self.data_frame.shape[0]} voices belonging to {self.num_classes} classes") 
  

3. 避免__getitem__访问空序列 #

当index超出__len__的范围时，切片会得到空序列，触发KeyError。

• 修复：在__getitem__开头添加边界判断：

  def __getitem__(self, index):
      if index >= self.__len__():
          raise IndexError("Index out of bounds")
      batch_x = self.data_frame["VoiceName"][index * self.batch_size:(index + 1) * self.batch_size]
      ......
  

4. 修正__get_voice的标签访问方式 #

batch_y是pandas Series时，直接用整数索引batch_y[i]可能出错，需按位置访问。

• 修复代码：

  def __get_voice(self,index, batch_x, batch_y, dir_data):
      X = []
      Y = []
      for i, name in enumerate(batch_x):
          label = batch_y.iloc[i]  # 改用iloc按位置取标签
          pathvoice = os.path.join(self.dir_data + '/'+ label+ '/'+ name)
          .....
  

内容的提问来源于stack exchange，提问作者Zara Nz