Python2.7下基于shutil的config.xml带结构备份及Git同步实现问询

Let's walk through a complete solution that handles all your requirements, plus a deep dive into using shutil efficiently for this task.

Complete Python 2.7 Script #

import os
import subprocess
import shutil
import datetime
import filecmp

def run_git_command(cmd, cwd=None):
    """Helper function to run git commands and handle errors"""
    try:
        subprocess.check_call(cmd, shell=True, cwd=cwd)
        return True
    except subprocess.CalledProcessError as e:
        print(f"Git command failed: {e}")
        return False

def main():
    # Configuration - Update these values to match your setup
    HOME_DIR = os.path.expanduser("~")
    BACKUP_REPO_URL = "https://your-backup-repo-url.git"  # Replace with your repo URL
    BACKUP_BASE_DIR = os.path.join(HOME_DIR, "Backup")
    SOURCE_BASE_DIR = os.path.join(HOME_DIR, "jobs")

    # Step 1: Clone or pull the backup repository
    if not os.path.exists(BACKUP_BASE_DIR):
        print("Cloning backup repository...")
        if not run_git_command(f"git clone {BACKUP_REPO_URL} {BACKUP_BASE_DIR}"):
            return
    else:
        print("Pulling latest changes from backup repository...")
        if not run_git_command("git pull", cwd=BACKUP_BASE_DIR):
            return

    # Step 2: Find all config.xml files under jobs/ directory
    config_files = []
    for root, dirs, files in os.walk(SOURCE_BASE_DIR):
        if "config.xml" in files:
            config_path = os.path.join(root, "config.xml")
            # Calculate relative path from HOME_DIR to preserve structure
            rel_path = os.path.relpath(config_path, HOME_DIR)
            config_files.append((config_path, rel_path))

    # Step 3: Copy files and handle git operations
    commit_needed = False
    commit_message = f"Backup config.xml files - {datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')}"

    for src_path, rel_path in config_files:
        dest_path = os.path.join(BACKUP_BASE_DIR, rel_path)
        dest_dir = os.path.dirname(dest_path)

        # Create destination directory if it doesn't exist
        if not os.path.exists(dest_dir):
            os.makedirs(dest_dir)
            print(f"Created directory: {dest_dir}")

        # Check if file needs to be copied
        file_needs_update = False
        if not os.path.exists(dest_path):
            file_needs_update = True
            print(f"New file detected: {rel_path}")
        elif not filecmp.cmp(src_path, dest_path, shallow=False):
            file_needs_update = True
            print(f"Modified file detected: {rel_path}")

        if file_needs_update:
            # Copy the file using shutil.copy2 to preserve metadata
            shutil.copy2(src_path, dest_path)
            print(f"Copied file: {src_path} -> {dest_path}")
            # Stage the file in git
            run_git_command(f"git add {rel_path}", cwd=BACKUP_BASE_DIR)
            commit_needed = True

    # Step 3.1/3.3: Commit and push if needed
    if commit_needed:
        print("Committing changes...")
        if run_git_command(f"git commit -m \"{commit_message}\"", cwd=BACKUP_BASE_DIR):
            print("Pushing changes to remote repository...")
            run_git_command("git push", cwd=BACKUP_BASE_DIR)
    else:
        print("No changes detected, skipping commit and push.")

if __name__ == "__main__":
    main()

Key Components Explained #

Git Operations #

• The run_git_command helper simplifies executing git commands and handling failures gracefully.
• We first check if the backup directory exists: if not, we clone the repo; if it does, we pull the latest changes to avoid merge conflicts.

File Discovery & Copying #

• os.walk traverses the jobs/ directory to find all config.xml files. We calculate the relative path from the home directory to preserve the original folder structure in the backup.
• We use filecmp.cmp (with shallow=False to check actual content, not just metadata) to skip copying unchanged files—this saves time and reduces unnecessary disk I/O.

Git Status Handling #

• We track if any files were added or modified with the commit_needed flag. Only when this flag is True do we run git commit and git push.
• New files are automatically staged with git add, while modified files are staged as soon as they're copied.

Efficient Usage of shutil Library #

Here's how to get the most out of shutil for this backup task:

• Use shutil.copy2 instead of shutil.copy: This function preserves critical file metadata (like creation/modification times), which is essential for maintaining accurate backup records.
• Pre-create directories: Always ensure the target directory exists before copying. In Python 2.7, os.makedirs doesn't have the exist_ok parameter, so we check for the directory first with os.path.exists to avoid errors.
• Avoid unnecessary copies: Use filecmp.cmp to compare source and target files before copying—this is a huge efficiency win, especially when dealing with large numbers of files that rarely change.
• Batch directory creation (optional): If you have hundreds of job directories, you could batch-create all target folders first before handling file copies, but for most Jenkins setups, the one-by-one approach is simple and sufficient.

内容的提问来源于stack exchange，提问作者Lakshmi narayana