如何解决Python生成器结合TensorFlow的‘generator already executing’错误？

问题原因 #

异常根源在于tf.data.Dataset.from_generator的使用方式错误：你直接传入了已实例化的生成器对象（train_gen、val_gen），但TensorFlow要求该参数是能创建新生成器实例的函数。AutoKeras在训练流程中（如数据解析、多轮训练、验证环节）会多次迭代数据集，同一个生成器实例无法被同时调用，因此触发此异常。

解决方案1：修改生成器传入逻辑（推荐） #

调整代码，让from_generator每次调用时都生成新的生成器实例：

1. 重构生成器创建函数 #

def create_generator_funcs(folder_path, batch_size, num_classes, validation_split=0.2):
    files = [os.path.join(folder_path, f) for f in os.listdir(folder_path) if f.endswith('.dat')]
    train_files, validation_files = train_test_split(files, test_size=validation_split)
    
    # 返回创建生成器的函数，每次调用生成新实例
    def train_gen_func():
        return data_generator(train_files, batch_size, num_classes)
    
    def val_gen_func():
        return data_generator(validation_files, batch_size, num_classes)
    
    return train_gen_func, val_gen_func

2. 修改数据集创建代码 #

if __name__ == "__main__":
    data_folder = r'/home/my_username/my_project_v2_original_data'
    train_gen_func, val_gen_func = create_generator_funcs(data_folder, BATCH_SIZE, NUM_OF_CLASSES)
    
    train_dataset = tf.data.Dataset.from_generator(
        train_gen_func,
        output_signature=(
            tf.TensorSpec(shape=(BATCH_SIZE, NUM_OF_FEATURES), dtype=tf.float32),
            tf.TensorSpec(shape=(BATCH_SIZE, NUM_OF_CLASSES), dtype=tf.float32)
        )
    )

    validation_dataset = tf.data.Dataset.from_generator(
        val_gen_func,
        output_signature=(
            tf.TensorSpec(shape=(BATCH_SIZE, NUM_OF_FEATURES), dtype=tf.float32),
            tf.TensorSpec(shape=(BATCH_SIZE, NUM_OF_CLASSES), dtype=tf.float32)
        )
    )

解决方案2：用类封装生成器（替代方案） #

通过类封装生成器逻辑，每次实例化类都会生成独立的生成器：

class DataGenerator:
    def __init__(self, file_paths, batch_size, num_classes):
        self.file_paths = file_paths
        self.batch_size = batch_size
        self.num_classes = num_classes
    
    def __call__(self):
        while True:
            for file_path in self.file_paths:
                x, y = load_data(file_path)
                for i in range(0, len(x), self.batch_size):
                    end = i + self.batch_size
                    batch_x = x[i:end]
                    batch_y = y[i:end]
                    if batch_x.shape[0] < self.batch_size:
                        padding = self.batch_size - batch_x.shape[0]
                        batch_x = np.pad(batch_x, ((0, padding), (0, 0)), mode='constant', constant_values=0)
                        batch_y = np.pad(batch_y, ((0, padding), (0, 0)), mode='constant', constant_values=0)
                    yield batch_x, batch_y

主函数中使用类生成器 #

if __name__ == "__main__":
    data_folder = r'/home/my_username/my_project_v2_original_data'
    files = [os.path.join(data_folder, f) for f in os.listdir(data_folder) if f.endswith('.dat')]
    train_files, validation_files = train_test_split(files, test_size=0.2)
    
    train_dataset = tf.data.Dataset.from_generator(
        DataGenerator(train_files, BATCH_SIZE, NUM_OF_CLASSES),
        output_signature=(
            tf.TensorSpec(shape=(BATCH_SIZE, NUM_OF_FEATURES), dtype=tf.float32),
            tf.TensorSpec(shape=(BATCH_SIZE, NUM_OF_CLASSES), dtype=tf.float32)
        )
    )

    validation_dataset = tf.data.Dataset.from_generator(
        DataGenerator(validation_files, BATCH_SIZE, NUM_OF_CLASSES),
        output_signature=(
            tf.TensorSpec(shape=(BATCH_SIZE, NUM_OF_FEATURES), dtype=tf.float32),
            tf.TensorSpec(shape=(BATCH_SIZE, NUM_OF_CLASSES), dtype=tf.float32)
        )
    )

额外优化建议 #

• 将output_signature中的shape从None改为固定的(BATCH_SIZE, ...)，因为生成器中已通过padding保证批次大小一致，能帮助TensorFlow优化数据处理流程。
• 在生成器中添加tf.convert_to_tensor，直接返回TensorFlow张量，减少数据转换开销。

内容的提问来源于stack exchange，提问作者user366312