如何编写脚本实现GCP Storage跨Bucket文件复制并重构下划线命名的目录结构

Hey there! Since you're new to GCP and gsutil, let's walk through a straightforward way to copy and restructure your files between buckets. This solution uses a simple bash script that handles the path splitting automatically.

Step 1: Verify Your gsutil Setup #

First, make sure you're authenticated to access your GCP buckets. Run this command if you haven't already:

gcloud auth login

Or if you prefer using gsutil directly:

gsutil config

You should also test that you can list files in your source bucket to confirm access:

gsutil ls gs://your-source-bucket-name

Step 2: Bash Script to Handle the Copy & Restructure #

Create a new file (e.g., restructure_bucket_files.sh) with the following script. Replace SOURCE_BUCKET and TARGET_BUCKET with your actual bucket names:

#!/bin/bash

# Replace these with your actual bucket names
SOURCE_BUCKET="gs://your-source-bucket-name"
TARGET_BUCKET="gs://your-target-bucket-name"

# Recursively list all files in the source bucket
gsutil ls "${SOURCE_BUCKET}/**" | while read -r source_file; do
    # Skip directories (gsutil ls includes both files and dirs)
    if [[ "$source_file" == */ ]]; then
        continue
    fi

    # Extract the relative path from the source file (remove bucket prefix)
    relative_path="${source_file#${SOURCE_BUCKET}/}"

    # Split the first directory part by underscore
    # Example: "Tests_111/tests.pdf" becomes dir1="Tests", dir2="111", rest="tests.pdf"
    IFS='_' read -r dir1 dir2 rest <<< "$(echo "$relative_path" | cut -d'/' -f1)"
    remaining_path="${relative_path#${dir1}_${dir2}/}"

    # Build the target file path
    target_file="${TARGET_BUCKET}/${dir1}/${dir2}/${remaining_path}"

    # Copy the file to the target bucket with the new structure
    echo "Copying ${source_file} to ${target_file}"
    gsutil cp -n "${source_file}" "${target_file}"
done

Step 3: How to Use the Script #

1. Save the script and make it executable:

chmod +x restructure_bucket_files.sh

2. Run the script:

./restructure_bucket_files.sh

Important Notes #

• The -n flag in gsutil cp prevents overwriting existing files in the target bucket. If you want to overwrite, remove this flag.
• This script assumes your source directories follow the Prefix_Number format (e.g., Tests_111, AllTests_111). If you have directories with more than one underscore, this will split on the first one—feel free to reach out if you need help adjusting for that!
• Test with a single file first to ensure it works as expected:

gsutil cp gs://your-source-bucket/Tests_111/tests.pdf gs://your-target-bucket/Tests/111/tests.pdf

• Make sure you have the necessary permissions: you need storage.objects.list on the source bucket and storage.objects.create on the target bucket.

内容的提问来源于stack exchange，提问作者Vicky