如何将自定义ImageNet子集（20类）转为Keras友好格式？

Got it, let's turn those TFRecords into a Keras-ready dataset that’s straightforward to work with. Here’s a practical, step-by-step approach tailored to your use case:

First, you need a function to decode the raw TFRecord examples into image-label pairs. This depends on how you structured your data when writing the TFRecords—adjust the feature description to match your actual saved fields:

import tensorflow as tf

def parse_tfrecord_example(example_proto):
    # Match this to the feature structure you used when creating TFRecords
    feature_description = {
        'image': tf.io.FixedLenFeature([], tf.string),  # If you saved image bytes
        'label': tf.io.FixedLenFeature([], tf.int64),    # Adjust type if labels are strings
    }
    
    # Parse the single example
    example = tf.io.parse_single_example(example_proto, feature_description)
    
    # Decode image bytes to a tensor (use decode_png if you saved PNGs)
    image = tf.io.decode_jpeg(example['image'], channels=3)
    image = tf.cast(image, tf.float32)  # Convert to float for preprocessing
    
    # Handle labels: uncomment below if you need one-hot encoding
    # label = tf.one_hot(example['label'], depth=20)
    label = example['label']
    
    return image, label

Next, load your TFRecords files and transform them into an optimized dataset that Keras can consume seamlessly:

# List all your TFRecords files (use glob to auto-find them if needed)
tfrecord_paths = ['path/to/train_data.tfrecords', 'path/to/val_data.tfrecords']

# Create base dataset from TFRecords
dataset = tf.data.TFRecordDataset(tfrecord_paths)

# Apply parsing function with parallel processing
dataset = dataset.map(parse_tfrecord_example, num_parallel_calls=tf.data.AUTOTUNE)

# Add preprocessing (adjust based on your model requirements)
def preprocess(image, label):
    # Resize images to match your input shape (e.g., 224x224 for ResNet)
    image = tf.image.resize(image, (224, 224))
    # Apply model-specific normalization (e.g., for ImageNet pre-trained models)
    image = tf.keras.applications.resnet50.preprocess_input(image)
    return image, label

dataset = dataset.map(preprocess, num_parallel_calls=tf.data.AUTOTUNE)

# Optimize for training: shuffle, batch, prefetch
BATCH_SIZE = 32
dataset = dataset.shuffle(buffer_size=1000)  # Adjust buffer size based on dataset size
dataset = dataset.batch(BATCH_SIZE)
dataset = dataset.prefetch(tf.data.AUTOTUNE)  # Overlap data loading and model execution

Your dataset is now fully Keras-compatible—you can pass it directly to model.fit(), model.evaluate(), or model.predict():

# Example: Initialize a model (use pre-trained or custom)
model = tf.keras.applications.ResNet50(weights=None, classes=20)

# Compile with appropriate loss/metrics
model.compile(
    optimizer=tf.keras.optimizers.Adam(),
    loss='sparse_categorical_crossentropy',  # Use 'categorical_crossentropy' if using one-hot labels
    metrics=['accuracy']
)

# Train directly on the dataset
model.fit(dataset, epochs=10)

• Feature Matching: Double-check that feature_description matches exactly how you wrote the TFRecords (e.g., if you saved image tensors instead of bytes, use FixedLenFeature([height, width, channels], tf.float32)).
• Label Handling: If your labels are string class names, parse them with tf.io.FixedLenFeature([], tf.string) and map them to integers using a lookup table.
• Train/Val Split: If you didn’t split your TFRecords into train/val subsets upfront, split the dataset like this:

  total_samples = sum(1 for _ in dataset.unbatch())
  train_size = int(0.8 * total_samples)
  train_dataset = dataset.take(train_size)
  val_dataset = dataset.skip(train_size)
  

• Performance: For large datasets, add dataset.cache() before shuffling to cache preprocessed data (use dataset.cache('/path/to/cache') if memory is limited).

内容的提问来源于stack exchange，提问作者Ayush Agarwal