GCP gsutil存储桶复制：如何跳过已复制文件缩短恢复耗时

Great question! I’ve dealt with this exact frustration before—those mandatory checks for existing files can turn a quick resume into a long wait, especially with large directories. Here are a few practical ways to skip those checks and only transfer new/unchanged files directly:

1. Use gsutil rsync (Recommended) #

The rsync command is purpose-built for syncing directories between local and GCS, and it’s optimized to minimize unnecessary checks by default. Unlike cp -rn, it compares file sizes and modification times locally and remotely to decide what to transfer—without scanning every existing file in detail when resuming.

Run this command to sync your local Downloads directory to the bucket:

gsutil rsync -r /Downloads gs://my-bucket/Downloads

• The -r flag enables recursive sync for subdirectories.
• If you want to completely skip any remote checks (note: this could lead to duplicate files if you’re not careful), you can combine it with a local filter to only transfer files created/modified after a specific date. For example:

  find /Downloads -type f -newermt "2024-05-01" | gsutil rsync -r -I gs://my-bucket/Downloads
  

  The -I flag tells rsync to read file paths from standard input, so it only processes the files you’ve filtered locally.

2. Filter Files Locally Before Transfer #

If you prefer sticking with cp, you can pre-filter the files you want to transfer locally (so gsutil never even sees the already copied ones). This avoids any remote checks entirely because you’re only sending new files.

Example: Transfer files modified after a specific date #

Use find to grab only files changed since your last transfer:

find /Downloads -type f -newermt "YYYY-MM-DD HH:MM:SS" | gsutil -m cp -I gs://my-bucket/Downloads/

Replace "YYYY-MM-DD HH:MM:SS" with the timestamp when you started your initial transfer.

Example: Maintain a local copy list #

For more control, keep a record of files you’ve already copied. After your first transfer, run:

find /Downloads -type f | sort > copied_files.txt

When resuming, use comm to find files not in the list:

comm -23 <(find /Downloads -type f | sort) <(sort copied_files.txt) | gsutil -m cp -I gs://my-bucket/Downloads/

This will only transfer files that weren’t in your original list, with zero remote checks.

Important Notes #

• Risk of duplicates: Skipping remote checks means gsutil won’t verify if a file already exists in the bucket. If you’ve modified a local file that’s already in GCS, this will overwrite it (unless you use -n with cp, but that brings back the check).
• rsync is safer: It balances speed with safety by default, so it’s the best option for most cases unless you’re absolutely sure you only need new files.

内容的提问来源于stack exchange，提问作者Chandan Thakur