Absolutely! You can absolutely exclude hidden files (those starting with .) and symbolic links when transferring directories with Paramiko and SCPClient—no rsync required. Let’s adjust your script to implement this, and even optimize it to skip the local temporary folder (which saves disk space and time).

Option 1: Direct Remote-to-Remote Transfer (Recommended) #

Instead of downloading files to your local machine first, you can transfer directly from target_host_1 to target_host_2 using Paramiko’s SFTP capability. This is faster and avoids cluttering your local disk. Here’s how to modify your script:

Step 1: Add SFTP Transfer Function #

Add this recursive function to handle directory traversal, filter out hidden files/symlinks, and transfer content directly:

def transfer_directory(sftp_src, sftp_dst, src_dir, dst_dir):
    """
    Recursively transfer files/dirs from src_dir (sftp_src) to dst_dir (sftp_dst),
    skipping hidden files and symbolic links.
    """
    # List all items in the source directory with attributes
    items = sftp_src.listdir_attr(src_dir)
    
    for item in items:
        item_name = item.filename
        
        # Skip hidden files (names starting with .)
        if item_name.startswith('.'):
            continue
        
        # Skip symbolic links (check S_IFLNK flag in file mode)
        if item.st_mode & 0o120000:
            continue
        
        src_path = os.path.join(src_dir, item_name)
        dst_path = os.path.join(dst_dir, item_name)
        
        if item.st_mode & 0o040000:
            # It's a directory: create it on destination if missing
            try:
                sftp_dst.stat(dst_path)
            except FileNotFoundError:
                sftp_dst.mkdir(dst_path)
            # Recurse into the subdirectory
            transfer_directory(sftp_src, sftp_dst, src_path, dst_path)
        else:
            # It's a regular file: transfer it directly
            with sftp_src.open(src_path, 'rb') as src_file:
                with sftp_dst.open(dst_path, 'wb') as dst_file:
                    dst_file.write(src_file.read())

Step 2: Replace Local Transfer Logic #

Remove the local folder creation, scp_1.get()/scp_2.put(), and shutil.rmtree() parts. Instead, use SFTP clients and call the transfer function:

# Open SFTP sessions for both target servers
sftp_1 = ssh_1.open_sftp()
sftp_2 = ssh_2.open_sftp()

# Delete old files on target_host_2 (more reliable than sleep)
command = "find /filepath -mindepth 1 -delete"
stdin, stdout, stderr = ssh_2.exec_command(command)
exit_status = stdout.channel.recv_exit_status()
if exit_status != 0:
    print(f"Warning: Failed to delete old backup files: {stderr.read().decode()}")

# Transfer directly from target_host_1 to target_host_2
transfer_directory(sftp_1, sftp_2, '/filepath/', '/filepath/')

# Close SFTP sessions
sftp_1.close()
sftp_2.close()

Option 2: Modified Local Transfer (If You Need to Keep Local Copy) #

If you still want to maintain a local temporary copy, adjust your script to filter files during download/upload:

Step 1: Get Filtered File List from Target 1 #

First, fetch the list of non-hidden, non-symlink files/directories from target_host_1:

def get_approved_files(ssh_client, remote_dir):
    """Fetch relative paths of files/dirs to transfer (exclude hidden, symlinks)"""
    # Use find command to list valid items
    find_cmd = f'''find "{remote_dir}" -type f ! -name ".*" ! -type l -printf "%P\\n"; 
                   find "{remote_dir}" -type d ! -name ".*" ! -type l -printf "%P\\n" | grep -v "^$"'''
    stdin, stdout, stderr = ssh_client.exec_command(find_cmd)
    return stdout.read().decode().splitlines()

remote_source_dir = '/filepath/'
local_target_dir = '/target_folder_local/'
files_to_transfer = get_approved_files(ssh_1, remote_source_dir)

Step 2: Download Only Approved Files #

Create the local directory structure and download each valid file:

# Create local directory structure
for item in files_to_transfer:
    local_path = os.path.join(local_target_dir, item)
    if os.path.basename(item):
        # It's a file: create parent directory
        os.makedirs(os.path.dirname(local_path), exist_ok=True)
    else:
        # It's a directory: create it
        os.makedirs(local_path, exist_ok=True)

# Download each file
for item in files_to_transfer:
    remote_path = os.path.join(remote_source_dir, item)
    local_path = os.path.join(local_target_dir, item)
    if os.path.isfile(local_path):  # Only download files (dirs already created)
        scp_1.get(remote_path, local_path)

Step 3: Upload Approved Files to Target 2 #

Upload the filtered local files to target_host_2:

# Upload to target_host_2
remote_dest_dir = '/filepath/'
for item in files_to_transfer:
    local_path = os.path.join(local_target_dir, item)
    remote_path = os.path.join(remote_dest_dir, item)
    if os.path.isfile(local_path):
        scp_2.put(local_path, remote_path)
    else:
        # Create directory on target if missing
        ssh_2.exec_command(f"mkdir -p '{remote_path}'")

Key Notes #

• Symlink Detection: We check the file mode flag 0o120000 (S_IFLNK) to identify symbolic links on the remote server.
• Hidden Files: We skip any item whose name starts with . (except . and .., which are handled automatically by directory traversal).
• Reliable Cleanup: Replaced time.sleep(5) with checking the command exit status to ensure old files are deleted before starting the backup.

内容的提问来源于stack exchange，提问作者Pheonix