无法从Amazon S3获取数据到EC2用于模型训练求助

Hey there! I remember being exactly where you are when I first started with AWS for deep learning—let’s break this down step by step so you can get your training up and running without the headache.

First things first: you need a way to connect your EC2 instance to S3. There are three main methods, depending on your workflow:

Method 1: AWS CLI (Most Beginner-Friendly) #

This is the easiest way to transfer files between S3 and EC2, and it’s great for one-time data syncs.

• Set Up IAM Permissions (Critical!): Don’t hardcode access keys—instead, attach an IAM role to your EC2 instance with S3 access. Go to the EC2 console, find your instance, navigate to "Actions > Security > Modify IAM role", and pick a role with permissions like AmazonS3ReadOnlyAccess (if you only need to read data) or a custom policy if you need to write back to S3. Trust me, using roles is way safer than storing keys locally.
• Install AWS CLI (If Needed): On Amazon Linux, it’s pre-installed. For Ubuntu/Debian, run:

  sudo apt update && sudo apt install aws-cli -y
  

• Test the Connection: Run this to list files in your bucket (replace your-bucket-name with your actual bucket):

  aws s3 ls s3://your-bucket-name
  

• Sync Data to EC2: To copy all your training data to a local directory on EC2 (way faster for training than reading directly from S3):

  aws s3 sync s3://your-bucket-name/path/to/training-data /home/ubuntu/local-data-folder
  

  The sync command skips already copied files, which is perfect for large datasets.

Method 2: Boto3 (Access Directly in Python Scripts) #

If you want to load data directly in your training code (without copying everything to EC2), use Boto3—the AWS Python SDK.

• Install Boto3:

  pip install boto3
  

• Example Code in Your Training Script:

  import boto3
  import pandas as pd
  from io import StringIO
  
  # Initialize S3 client (uses your EC2 instance's IAM role automatically)
  s3 = boto3.client('s3')
  
  # Read a CSV file directly from S3 into a pandas DataFrame
  response = s3.get_object(Bucket='your-bucket-name', Key='data/train/labels.csv')
  csv_content = response['Body'].read().decode('utf-8')
  df = pd.read_csv(StringIO(csv_content))
  
  # Download a single image to use in your dataset
  s3.download_file('your-bucket-name', 'data/train/img001.jpg', '/tmp/train/img001.jpg')
  

Method 3: Mount S3 as a Local Directory (s3fs) #

If you want to treat S3 like a regular folder on your EC2 instance (great for small datasets or when you don’t want to copy files), use s3fs:

• Install s3fs:

  sudo apt install s3fs -y  # Ubuntu/Debian
  sudo yum install s3fs-fuse -y  # Amazon Linux
  

• Mount the Bucket:

  # Create a mount point
  sudo mkdir /mnt/s3-data
  # Mount using your EC2 instance's IAM role (no keys needed!)
  sudo s3fs your-bucket-name /mnt/s3-data -o iam_role=auto
  

  Now you can access your S3 files like any local folder: cd /mnt/s3-data

Once you have access to your data, integrating it into your deep learning workflow is straightforward:

If You Copied Data to EC2 Local Storage #

Just point your training script to the local directory. For example, in PyTorch:

from torchvision.datasets import ImageFolder

train_dataset = ImageFolder('/home/ubuntu/local-data-folder/train')
# Rest of your training code (dataloader, model, optimizer, etc.)...

Then run your script from the terminal:

python train.py --epochs 10 --batch-size 32

If Using Boto3 Directly #

Create a custom dataset class that loads data from S3 on the fly. This is great for very large datasets that don’t fit on EC2’s storage:

from torch.utils.data import Dataset
from PIL import Image

class S3ImageDataset(Dataset):
    def __init__(self, s3_bucket, s3_prefix, transform=None):
        self.s3 = boto3.client('s3')
        self.bucket = s3_bucket
        # List all files in the S3 prefix
        self.files = [obj['Key'] for obj in self.s3.list_objects_v2(Bucket=bucket, Prefix=s3_prefix)['Contents']]
        self.transform = transform

    def __len__(self):
        return len(self.files)

    def __getitem__(self, idx):
        # Download image to a temporary file
        key = self.files[idx]
        tmp_path = f'/tmp/{key.split("/")[-1]}'
        self.s3.download_file(self.bucket, key, tmp_path)
        # Load and transform the image
        img = Image.open(tmp_path)
        if self.transform:
            img = self.transform(img)
        return img

If You Mounted S3 as a Local Folder #

Treat /mnt/s3-data like any other directory in your script. For example, in TensorFlow:

import tensorflow as tf

train_ds = tf.keras.utils.image_dataset_from_directory(
    '/mnt/s3-data/train',
    image_size=(224, 224),
    batch_size=32
)

• Pick the Right EC2 Instance: Use a GPU instance like g4dn.xlarge or p3.2xlarge for deep learning—CPU instances will be way too slow for most training tasks.
• Use Fast Storage: If you’re copying data to EC2, attach a gp3 EBS volume or use the local NVMe storage on GPU instances (it’s much faster than the default root volume).
• Check Permissions: If you get "Access Denied" errors, double-check your IAM role permissions and your S3 bucket’s access policy (make sure it allows your EC2 instance’s role to access it).

内容的提问来源于stack exchange，提问作者vidit02100