如何用Python和boto3上传系统文件至AWS？已完成前置配置但卡连接

Hey there! Let's get your Python script connected to AWS and uploading those system files smoothly. Since you already have the core setup (AWS account, EC2 instance, S3 bucket, Python/boto3/AWS CLI installed), we'll focus on fixing the connection hurdle and implementing the upload logic.

Step 1: Verify AWS CLI Credentials (Critical for Boto3) #

Boto3 automatically pulls credentials from your AWS CLI config, so let's make sure that's squared away first:

• Run this terminal command to check your existing configuration:

  aws configure list
  

• If you see empty values for access_key or secret_key, run aws configure and input:
  • Your AWS Access Key ID
  • Your AWS Secret Access Key
  • The region where your S3 bucket is hosted (e.g., us-east-1)
  • Default output format (you can leave this as json)

Pro tip: If you're running this on your EC2 instance, skip access keys entirely and attach an IAM Role with AmazonS3FullAccess (or restricted permissions for just your target bucket). This is way more secure and avoids hardcoding sensitive credentials.

Step 2: Test Boto3 Connection #

Let's write a quick test script to confirm Python can talk to AWS. Save this as test_connection.py:

import boto3

try:
    # Initialize S3 client
    s3 = boto3.client('s3')
    # List your S3 buckets to verify connection
    response = s3.list_buckets()
    print("Connected successfully! Your buckets:")
    for bucket in response['Buckets']:
        print(f"- {bucket['Name']}")
except Exception as e:
    print(f"Connection failed: {str(e)}")

Run it with python test_connection.py. If it lists your bucket, you're ready to move on! If not, double-check your credentials, region, or IAM permissions.

Step 3: Upload a Single System File to S3 #

Once connected, here's a straightforward script to upload a single system file (like /var/log/syslog or a config file):

import boto3

def upload_file_to_s3(local_file_path, bucket_name, s3_file_name):
    s3 = boto3.client('s3')
    try:
        s3.upload_file(local_file_path, bucket_name, s3_file_name)
        print(f"Successfully uploaded {local_file_path} to s3://{bucket_name}/{s3_file_name}")
        return True
    except FileNotFoundError:
        print(f"Error: The file {local_file_path} was not found")
        return False
    except Exception as e:
        print(f"Upload failed: {str(e)}")
        return False

# Example usage
upload_file_to_s3(
    local_file_path='/path/to/your/system/file.txt',
    bucket_name='your-existing-s3-bucket',
    s3_file_name='uploaded_system_files/backup.txt'  # Path/name for the file in S3
)

Step 4: Upload an Entire System Folder (Optional) #

If you need to upload multiple files from a directory (like a logs folder), use this script to traverse the folder and upload each file:

import boto3
import os

def upload_folder_to_s3(local_folder_path, bucket_name, s3_folder_prefix):
    s3 = boto3.client('s3')
    for root, dirs, files in os.walk(local_folder_path):
        for file in files:
            local_file_path = os.path.join(root, file)
            # Create a relative path for the file in S3
            s3_file_path = os.path.join(s3_folder_prefix, os.path.relpath(local_file_path, local_folder_path))
            try:
                s3.upload_file(local_file_path, bucket_name, s3_file_path)
                print(f"Uploaded: {local_file_path} -> s3://{bucket_name}/{s3_file_path}")
            except Exception as e:
                print(f"Failed to upload {local_file_path}: {str(e)}")

# Example usage
upload_folder_to_s3(
    local_folder_path='/var/log',
    bucket_name='your-existing-s3-bucket',
    s3_folder_prefix='system_log_backups/'  # Prefix to group files in S3
)

Key Notes to Avoid Headaches #

• Permissions: Make sure your AWS credentials (or EC2 IAM role) have the s3:PutObject permission for your target bucket.
• Large Files: For files over 10GB, boto3 automatically handles multipart uploads via upload_file—no extra code needed!
• Absolute Paths: For system files, always use absolute paths (like /var/log/syslog) instead of relative paths to avoid "file not found" errors.

内容的提问来源于stack exchange，提问作者Neeraja Kl