Hey there! Let's work through how to integrate your custom preprocessing into PyTorch's DataLoader pipeline, and fix a few small bugs in your code along the way.

First, let's get straight to the core question: You should use ImageFolder to load your dataset first, then apply your preprocessing steps as part of PyTorch's transform pipeline. Preprocessing all images upfront and saving them to disk is usually not ideal (wastes storage, makes parameter adjustments a hassle later), so we'll focus on the flexible real-time transform approach.

Step 1: Fix Your Preprocessing Functions #

Your current code has a few bugs that will cause errors if run as-is — let's correct those first:

• In gaussian_blur, you named the input parameter img but reference image inside the function, plus you're returning the blurred image instead of the required N = I - G result.
• In normalise, there are typos (imgs_std → img_std, inconsistent variable naming between img_normalised and img_normalized), and the loop over img.shape[1] targets the wrong dimension (width instead of channels/height).

Here's the cleaned-up, functional version:

import cv2
import numpy as np
from PIL import Image
import torch
import torchvision.transforms as transforms

def gaussian_blur(img):
    # Convert PIL Image to numpy array (cv2 works with numpy)
    img_np = np.array(img).astype(np.float32)
    # Apply Gaussian blur: kernel (65,65), sigma=10
    blurred = cv2.GaussianBlur(img_np, (65, 65), 10)
    # Compute normalized image N = I - G
    normalized_img = img_np - blurred
    return normalized_img

def normalise(img_np):
    # First normalization: subtract global mean, divide by global std
    img_mean = np.mean(img_np)
    img_std = np.std(img_np)
    img_normalized = (img_np - img_mean) / img_std
    
    # Second normalization per channel (handles grayscale or color images)
    if len(img_np.shape) == 3:  # Color image (H, W, C)
        for channel in range(img_np.shape[2]):
            chan_mean = np.mean(img_normalized[:, :, channel])
            chan_std = np.std(img_normalized[:, :, channel])
            img_normalized[:, :, channel] = (img_normalized[:, :, channel] - chan_mean) / chan_std
    else:  # Grayscale image (H, W)
        chan_mean = np.mean(img_normalized)
        chan_std = np.std(img_normalized)
        img_normalized = (img_normalized - chan_mean) / chan_std
    
    return img_normalized

Step 2: Wrap Preprocessing as PyTorch Transforms #

PyTorch's transform system expects functions/classes that take a PIL Image (or tensor) and return a processed tensor or image. We'll create custom transform classes to wrap our preprocessing logic:

class GaussianBlurTransform:
    def __call__(self, img):
        return gaussian_blur(img)

class NormaliseTransform:
    def __call__(self, img_data):
        # Handle cases where input might be a tensor (from early ToTensor() calls)
        if isinstance(img_data, torch.Tensor):
            img_data = img_data.numpy().transpose(1, 2, 0)  # Convert (C, H, W) tensor to (H, W, C) numpy
        return normalise(img_data)

# Compose all transforms into a single pipeline
custom_transform = transforms.Compose([
    GaussianBlurTransform(),
    NormaliseTransform(),
    transforms.ToTensor()  # Convert final numpy array to PyTorch tensor (C, H, W)
])

Step 3: Integrate with ImageFolder and DataLoader #

Now we can plug our custom transform into ImageFolder and wrap it in a DataLoader for training:

from torchvision.datasets import ImageFolder
from torch.utils.data import DataLoader

# Assume your dataset follows the ImageFolder structure:
# dataset_root/
#     class_1/
#         img1.jpg
#         img2.jpg
#     class_2/
#         img1.jpg
#         ...

# Load dataset with our preprocessing pipeline
dataset = ImageFolder(root="path/to/your/dataset_root", transform=custom_transform)

# Create DataLoader for batching/shuffling
dataloader = DataLoader(
    dataset,
    batch_size=32,
    shuffle=True,
    num_workers=4  # Parallelize preprocessing across CPU cores
)

Why Not Preprocess Images First? #

Preprocessing all images and saving them to disk is technically possible, but has major downsides:

• Storage waste: Floating-point preprocessed images take up far more space than compressed JPEG/PNG files.
• Infleixbility: If you later want to adjust the Gaussian kernel size, sigma, or normalization logic, you'll have to reprocess every single image.
• No dynamic augmentation: Adding data augmentation (like random cropping, flipping) later becomes much harder with pre-saved images.

The transform pipeline approach avoids all these issues, and PyTorch's num_workers parameter offsets any computational overhead by parallelizing preprocessing.

Quick Validation Tip #

Always test a single batch to make sure your preprocessing works as expected:

for imgs, labels in dataloader:
    print(f"Batch shape: {imgs.shape}")
    print(f"Batch mean: {imgs.mean():.4f}, Batch std: {imgs.std():.4f}")
    break

内容的提问来源于stack exchange，提问作者Beginner