本地文件夹与AWS S3桶内容比对及差异化加密上传需求

Hey there! Let's walk through a solid, reliable solution for your encrypted S3 sync workflow. The key here is balancing three things: comparing local vs S3 content, securely encrypting files before upload, and tracking what's already been processed to avoid redundant work. Here's how to build it:

We'll use a combination of AWS CLI for S3 interactions, OpenSSL for encryption, and a local tracking file to keep tabs on which files have been successfully encrypted and uploaded. This setup ensures:

• Only unprocessed local files get encrypted
• Encrypted copies are deleted locally only after confirmed upload
• Daily runs automatically pick up new files without redoing work

1. Prerequisites #

First, make sure you have these set up:

• AWS CLI installed and configured with permissions to read/write your target S3 bucket (run aws configure to set credentials)
• OpenSSL installed (pre-installed on most Linux/macOS systems)
• A secure way to store your encryption password (never hardcode it in scripts!)

2. Build the Sync Script #

Create a shell script (e.g., s3_encrypted_sync.sh) with the following logic. I'll break down each section:

2.1 Define Configuration Variables #

Start by setting up your paths and secrets (use environment variables for sensitive data!):

#!/bin/bash

# Local folder containing images to process
LOCAL_DIR="/path/to/your/local/image/folder"
# Target S3 bucket path (include trailing slash)
S3_BUCKET="s3://your-bucket-name/encrypted-images/"
# Local file to track processed files (stores unique file signatures)
TRACK_FILE="$LOCAL_DIR/.processed_files"
# Encryption password (pull from environment variable for security)
ENCRYPT_PASS="${S3_ENCRYPT_PASS:-}"

# Exit if password isn't set
if [ -z "$ENCRYPT_PASS" ]; then
    echo "Error: S3_ENCRYPT_PASS environment variable not set"
    exit 1
fi

2.2 Initialize Tracking File #

If the tracking file doesn't exist yet, create it to avoid errors:

if [ ! -f "$TRACK_FILE" ]; then
    touch "$TRACK_FILE"
    echo "Initialized new tracking file: $TRACK_FILE"
fi

2.3 Process Unencrypted/Unuploaded Files #

Loop through local image files, check if they've already been processed, and handle encryption/upload:

# Find all image files (adjust extensions to match your needs)
find "$LOCAL_DIR" -type f \( -iname "*.jpg" -o -iname "*.png" -o -iname "*.gif" \) | while read -r FILE; do
    # Generate a unique signature for the file (path + last modified time)
    # This ensures we reprocess files if they're updated locally
    FILE_SIG=$(echo "$FILE$(stat -c %Y "$FILE")" | md5sum | cut -d' ' -f1)
    
    # Skip if file is already in our tracking list
    if grep -q "$FILE_SIG" "$TRACK_FILE"; then
        continue
    fi

    echo "Starting processing for: $FILE"
    
    # Step 1: Encrypt the local file
    ENCRYPTED_FILE="$FILE.enc"
    openssl enc -aes-256-cbc -salt -in "$FILE" -out "$ENCRYPTED_FILE" -pass pass:"$ENCRYPT_PASS"
    
    # Step 2: Upload encrypted file to S3
    aws s3 cp "$ENCRYPTED_FILE" "$S3_BUCKET$(basename "$ENCRYPTED_FILE")"
    
    # Step 3: Verify upload success before cleaning up
    if aws s3 ls "$S3_BUCKET$(basename "$ENCRYPTED_FILE")" >/dev/null 2>&1; then
        # Delete local encrypted copy
        rm "$ENCRYPTED_FILE"
        # Record the processed file signature to avoid rework
        echo "$FILE_SIG|$FILE|$(basename "$ENCRYPTED_FILE")" >> "$TRACK_FILE"
        echo "Successfully processed and uploaded: $FILE"
    else
        echo "Warning: Upload failed for $FILE. Keeping encrypted copy at $ENCRYPTED_FILE"
    fi
done

2.4 Optional: Sync Deletions (If Needed) #

If you want to reprocess files that were deleted from S3, add this section to check existing tracking records against S3:

# Check if tracked files still exist in S3 (optional)
while IFS='|' read -r SIG LOCAL_FILE S3_FILENAME; do
    if ! aws s3 ls "$S3_BUCKET$S3_FILENAME" >/dev/null 2>&1; then
        echo "S3 file missing: $S3_FILENAME. Reprocessing local file: $LOCAL_FILE"
        # Remove the signature from tracking so it gets reprocessed
        sed -i "/^$SIG/d" "$TRACK_FILE"
    fi
done < "$TRACK_FILE"

3. Key Security & Reliability Notes #

• Password Safety: Always use an environment variable (like export S3_ENCRYPT_PASS="your-strong-password") instead of hardcoding. For production, consider using AWS Secrets Manager to fetch the password dynamically.
• File Signatures: Using the file path + modified time ensures we don't skip files that have been updated locally. If you only care about new files (not updates), you can adjust the signature to just the file path.
• Error Handling: The script verifies upload success before deleting the encrypted copy and updating the tracking file—this prevents data loss if the upload fails.

4. Automate Daily Execution #

To run this script automatically every day (e.g., at 2 AM), use cron:

1. Open your crontab editor: crontab -e
2. Add this line (adjust the script path and log path to match your setup):

0 2 * * * /path/to/s3_encrypted_sync.sh >> /var/log/s3_sync.log 2>&1

This will run the script daily and log output to help with debugging.

内容的提问来源于stack exchange，提问作者spesmagna