Hey there! I totally get the hassle of dealing with restricted internet when trying to work with PyTorch datasets—been in your shoes before. Let’s walk through your questions clearly:

There’s no strict rule enforced by PyTorch, but following a consistent, organized structure will save you a lot of headaches later. Here’s what I recommend:

• Create a dedicated data folder in your project’s root directory. This keeps your dataset files separate from your code, making the project cleaner and easier to share.
• Inside data, create subfolders named after the dataset (e.g., ./data/cifar10, ./data/mnist) to avoid mixing up different datasets.
• For datasets split into train/test sets, add further subfolders like ./data/cifar10/train and ./data/cifar10/test if needed.

This aligns with most PyTorch tutorial conventions, so you won’t have to adjust paths much if you follow existing code examples.

It depends on whether you’re using a built-in PyTorch dataset or a custom one:

Case 1: Using PyTorch’s Built-in Dataset Classes #

Most common datasets (like MNIST, CIFAR-10, ImageNet) have pre-defined classes in torchvision.datasets. These classes accept a root parameter (where your dataset lives) and a download=False flag to skip automatic downloading.

Important: You need to match the folder structure that PyTorch expects from its official downloads. For example, MNIST expects a raw folder with the original .idx files and a processed folder with the serialized tensors, or you can place the extracted files directly in the root folder (PyTorch usually detects this automatically).

Here’s an example for MNIST:

from torchvision.datasets import MNIST
from torchvision.transforms import ToTensor

# Assume you've extracted the MNIST files to ./data/mnist
train_dataset = MNIST(
    root="./data/mnist",
    train=True,
    download=False,  # *Critical*: Disable auto-download
    transform=ToTensor()
)

Case 2: Creating a Custom Dataset #

For your own datasets (like personal images, CSV data, etc.), you’ll need to define a custom class that inherits from torch.utils.data.Dataset and implements two core methods: __len__ (returns the number of samples) and __getitem__ (loads a single sample by index).

Here’s a simple example for an image dataset where images are stored in a folder and labels are in filenames:

import os
from PIL import Image
from torch.utils.data import Dataset
from torchvision.transforms import ToTensor

class CustomImageDataset(Dataset):
    def __init__(self, img_dir, transform=None):
        self.img_dir = img_dir
        self.transform = transform
        # Extract labels from filenames (adjust this based on your dataset's structure)
        self.img_files = [f for f in os.listdir(img_dir) if f.endswith(('.png', '.jpg'))]
        self.labels = [os.path.splitext(f)[0].split('_')[1] for f in self.img_files]  # Example: "cat_001.jpg" → "cat"

    def __len__(self):
        return len(self.img_files)

    def __getitem__(self, idx):
        img_path = os.path.join(self.img_dir, self.img_files[idx])
        image = Image.open(img_path).convert("RGB")  # Ensure consistent color format
        label = self.labels[idx]
        
        # Convert label to numerical value if needed (e.g., "cat" → 0, "dog" → 1)
        # label = self.label_to_idx[label]
        
        if self.transform:
            image = self.transform(image)
        return image, label

# Usage
train_dataset = CustomImageDataset(
    img_dir="./data/my_custom_images/train",
    transform=ToTensor()
)

• Match the expected folder structure: When using built-in datasets, double-check that your local files mirror the structure PyTorch downloads. If you’re unsure, run the dataset class with download=True once on a network-enabled device to see the structure, then replicate it locally.
• Verify dataset integrity: After transferring files, check that no files are corrupted or missing. Compare file sizes or MD5 hashes with the official dataset’s values if available—this prevents weird errors during training.
• Use relative paths: Stick to relative paths (like ./data/xxx) instead of absolute paths (e.g., C:/Users/MyName/data/xxx). This makes your project portable across different machines.
• Preprocess consistently: Apply the same transforms (resizing, normalization, etc.) that you would use with the downloaded dataset. Inconsistent preprocessing can ruin model performance.
• DataLoader efficiency: For large datasets, use torch.utils.data.DataLoader with an appropriate num_workers value to speed up data loading. Note that on Windows, setting num_workers=0 can avoid multiprocessing bugs.

内容的提问来源于stack exchange，提问作者NetFre