如何用rsync对本地100GB大文件实现5分钟增量备份？

Great question—dealing with 100GB files and frequent incremental backups without slow diff checks is exactly where rsync shines. Its block-level comparison is way faster than line-by-line diff, and we can combine it with hard links to manage multiple timestamped backups efficiently. Here’s a step-by-step solution:

Core Concept #

Instead of recreating the entire file every 5 minutes, we’ll:

• Use hard links to instantly "clone" the previous backup (no disk space wasted, no copy time)
• Run rsync to update only the changed blocks in the new backup file
• Name each backup with the exact timestamp of when the backup ran (matching your example.txt.HH:MM format)

Step 1: Create a Backup Script #

Save this as backup_example.sh (make it executable with chmod +x backup_example.sh):

#!/bin/bash

# Source file path
SOURCE="/path/to/example.txt"
# Backup directory (create it first with mkdir -p /path/to/backups)
BACKUP_DIR="/path/to/backups"
# Get current timestamp in HH:MM format
TIMESTAMP=$(date +"%H:%M")
# Full path to the new backup file
NEW_BACKUP="${BACKUP_DIR}/example.txt.${TIMESTAMP}"
# Path to the most recent backup (we'll use this as a reference)
LAST_BACKUP=$(ls -t "${BACKUP_DIR}/example.txt."* 2>/dev/null | head -n 1)

# Handle first backup (no previous backup exists)
if [ -z "$LAST_BACKUP" ]; then
    echo "First backup: copying entire file..."
    rsync -a --inplace "$SOURCE" "$NEW_BACKUP"
else
    echo "Incremental backup: using $LAST_BACKUP as reference..."
    # Create a hard link to the last backup (instant, no extra space)
    ln "$LAST_BACKUP" "$NEW_BACKUP"
    # Sync source to the new backup: rsync will only update changed blocks
    rsync -a --inplace "$SOURCE" "$NEW_BACKUP"
fi

# Optional: Keep only the last 24 backups (adjust as needed)
# NUM_BACKUPS_TO_KEEP=24
# ls -t "${BACKUP_DIR}/example.txt."* | tail -n +$((NUM_BACKUPS_TO_KEEP + 1)) | xargs rm -f

echo "Backup completed: $NEW_BACKUP"

Step 2: Schedule the Script with Cron #

To run the backup every 5 minutes, edit your crontab with crontab -e and add this line:

*/5 * * * * /path/to/backup_example.sh >> /path/to/backups/backup_log.txt 2>&1

This will execute the script every 5 minutes, log output to backup_log.txt, and generate timestamped backups like example.txt.00:05, example.txt.00:10, etc., as you requested.

Key Advantages Over Diff #

• Speed: Rsync uses block-level checksums instead of line-by-line comparison, so it’s drastically faster for large files with small changes.
• Space Efficiency: Hard links mean each backup only uses disk space for the parts of the file that changed—no redundant copies of unchanged data.
• Simplicity: The script handles both initial full backups and subsequent incrementals automatically.

Notes #

• Ensure the backup directory has enough disk space: each incremental backup will only take up space equal to the changes made since the last backup.
• The --inplace flag tells rsync to modify the target file in place, which is crucial for large files (avoids creating a full temporary copy).
• To restore a specific version, just copy the corresponding example.txt.HH:MM file back to your original path.

内容的提问来源于stack exchange，提问作者mohit