GCP云存储Search & Conversation Data Store数据清空失败求助

问题描述 #

需要清空GCP Search & Conversation Data Store App的所有数据以便删除应用，但使用google-cloud-storage库在JupyterLab Notebook执行删除代码时，控制台提示Blob已成功删除，但数据仍存在。调整if语句、添加try/except块后问题依旧，相关代码如下：

from google.cloud import storage

def purge_gcs_data(bucket_name, folder):
    storage_client = storage.Client()
    bucket = storage_client.bucket(bucket_name)
    print(f"Storage bucket is '{bucket.name}'")

    # Delete individual record files
    record_blobs = bucket.list_blobs(prefix=f"{folder}/")
    for blob in record_blobs:
        print(f"Deleting {blob.name}...")
        blob.delete()

    # Delete the metadata file
    metadata_blob = bucket.blob(f"{folder}/metadata.jsonl")
    if metadata_blob.exists():
        print("Deleting metadata.jsonl...")
        metadata_blob.delete()

    # Check if the folder is now empty
    remaining_blobs = list(bucket.list_blobs(prefix=f"{folder}/"))
    if not remaining_blobs:
        print(f"Confirmation: All objects in '{folder}' of bucket '{bucket_name}' have been successfully purged.")
    else:    
        for blob in remaining_blobs:
            print(blob.name)
    
            try:
                remaining_blobs = list(bucket.list_blobs(prefix=f"{folder}/"))
                if not remaining_blobs:
                    print(f"Confirmation: All objects in '{folder}' of bucket '{bucket_name}' have been successfully purged.")
                else:
                    print("Warning: Some objects were not deleted:")
                for blob in remaining_blobs:
                    print(f"Remaining Object: {blob.name}")
            except Exception as e:
                print(f"Error during fetching remaining blobs: {e}")

# Call the function 
PROJECT_ID_SUFFIX = PROJECT_ID.split("-")[-1]
GCS_BUCKET = f"shared-aif-bucket-{PROJECT_ID_SUFFIX}"
GCS_FOLDER = "clinical note summarization demo"

purge_gcs_data(GCS_BUCKET, GCS_FOLDER)

代码问题分析 #

原代码存在几个关键逻辑问题：

• 缩进错误：检查剩余Blob的try/except块被嵌套在遍历剩余Blob的循环内部，导致每遍历一个Blob就重复执行检查逻辑，判断逻辑混乱。
• 迭代器遍历隐患：直接遍历list_blobs返回的迭代器时，删除操作可能导致迭代器无法完整遍历所有Blob，出现漏删情况。
• 冗余操作：metadata.jsonl已经被包含在前面的record_blobs遍历中，后续单独删除属于重复操作。

修正后的代码 #

from google.cloud import storage

def purge_gcs_data(bucket_name, folder):
    storage_client = storage.Client()
    bucket = storage_client.bucket(bucket_name)
    print(f"Storage bucket is '{bucket.name}'")

    # 获取目标文件夹下所有Blob（含子文件夹），转成列表避免迭代器遍历隐患
    all_blobs = list(bucket.list_blobs(prefix=f"{folder}/"))
    
    # 批量删除所有Blob，比单个删除效率更高
    if all_blobs:
        print(f"正在删除 {len(all_blobs)} 个对象...")
        bucket.delete_blobs(all_blobs)
    else:
        print(f"'{folder}' 下未找到任何对象")

    # 重新检查剩余Blob，确认删除结果
    try:
        remaining_blobs = list(bucket.list_blobs(prefix=f"{folder}/"))
        if not remaining_blobs:
            print(f"确认：Bucket '{bucket_name}' 中 '{folder}' 下的所有对象已成功删除。")
        else:
            print("警告：仍有部分对象未删除：")
            for blob in remaining_blobs:
                print(f"剩余对象：{blob.name}")
    except Exception as e:
        print(f"检查剩余对象时出错：{e}")

# 调用函数（确保PROJECT_ID已在Notebook中提前定义）
PROJECT_ID_SUFFIX = PROJECT_ID.split("-")[-1]
GCS_BUCKET = f"shared-aif-bucket-{PROJECT_ID_SUFFIX}"
GCS_FOLDER = "clinical note summarization demo"

purge_gcs_data(GCS_BUCKET, GCS_FOLDER)

额外注意事项 #

• 权限验证：确保JupyterLab使用的服务账号拥有GCS Bucket的storage.objects.delete权限，无权限会导致删除操作静默失败。
• 服务端缓存：若删除GCS数据后，Search & Conversation Data Store仍显示数据，可能是服务端缓存，等待几分钟后刷新页面查看即可。
• 虚拟文件夹特性：GCS中没有真实的文件夹结构，删除所有Blob后，“文件夹”会自动消失，无需单独删除文件夹对象。

内容的提问来源于stack exchange，提问作者medako