基于Google Cloud SDK 204.0.0查询GCP项目中所有数据集、数据表及存储桶总大小的方法咨询

Got it, since you're stuck on the older Google Cloud SDK 204.0.0, here's how you can get the total sizes you need without upgrading. I'll break this down into BigQuery (datasets/tables) and Cloud Storage buckets:

BigQuery: Total Size of All Datasets & Tables #

You're right that a dataset's total size is just the sum of all its tables' sizes. Since SDK 204.0.0 uses the legacy bq tool (or gcloud bq wrapper), we can use a bash script to iterate through all datasets and tables, sum their sizes:

Step-by-Step Script #

Replace your-project-id with your actual GCP project ID:

PROJECT_ID="your-project-id"
total_bq_bytes=0

# Fetch all datasets in the project
datasets=$(bq ls --project_id=$PROJECT_ID --format=csv | tail -n +2 | cut -d',' -f1)

for dataset in $datasets; do
    # Fetch all tables in the current dataset
    tables=$(bq ls --project_id=$PROJECT_ID --format=csv $dataset | tail -n +2 | cut -d',' -f1)
    for table in $tables; do
        # Extract table size in bytes (ignores views, which have 0 bytes)
        table_size=$(bq show --project_id=$PROJECT_ID --format=json $dataset.$table | grep -Po '"numBytes": "\K\d+')
        total_bq_bytes=$((total_bq_bytes + table_size))
    done
done

# Convert to human-readable format (MB, adjust to GB/TB as needed)
echo "Total BigQuery data size: $((total_bq_bytes / 1024 / 1024)) MB ($total_bq_bytes bytes)"

Notes: #

• This script skips views automatically because their numBytes value is 0.
• You'll need the BigQuery Data Viewer IAM permission to run these commands.
• If you prefer using gcloud bq instead of the standalone bq tool, just replace bq with gcloud bq in the script.

Cloud Storage: Total Size of All Buckets #

For Cloud Storage, SDK 204.0.0 relies on gsutil (since the newer gcloud storage commands weren't available yet). Use this script to sum up all bucket sizes:

Step-by-Step Script #

total_gcs_bytes=0

# Fetch all buckets in the project
buckets=$(gsutil ls)

for bucket in $buckets; do
    # Get total size of the bucket (use `-a` flag to include versioned objects if needed)
    bucket_size=$(gsutil du -s $bucket | awk '{print $1}')
    total_gcs_bytes=$((total_gcs_bytes + bucket_size))
done

# Convert to human-readable format
echo "Total Cloud Storage size: $((total_gcs_bytes / 1024 / 1024)) MB ($total_gcs_bytes bytes)"

Notes: #

• Add the -a flag to gsutil du -s (like gsutil du -s -a $bucket) if you want to include size from versioned objects and deleted objects with retention policies.
• You'll need the Storage Viewer IAM permission to access bucket size details.

内容的提问来源于stack exchange，提问作者user15110035