如何高效批量下载Azure Blob Storage中指定前缀的文件？

• 用前缀过滤替代通配符
  Azure Blob Storage不支持*这类通配符匹配，但提供了前缀过滤功能。你需要构造精确的前缀字符串ID-YYYYMMDD-，通过这个前缀直接筛选目标Blob，无需遍历全量文件列表。

• Python代码实现示例
  使用azure-storage-blob库可以快速实现前缀过滤和批量下载：

  from azure.storage.blob import BlobServiceClient
  import os
  
  # 初始化Blob服务客户端
  connection_string = "你的存储账户连接字符串"
  container_name = "目标容器名称"
  blob_service_client = BlobServiceClient.from_connection_string(connection_string)
  container_client = blob_service_client.get_container_client(container_name)
  
  # 构造匹配前缀（替换为你的ID和日期）
  target_prefix = "123-20240520-"
  
  # 获取所有符合前缀的Blob
  matching_blobs = container_client.list_blobs(name_starts_with=target_prefix)
  
  # 创建下载目录（如果不存在）
  download_dir = "./azure_blob_downloads"
  os.makedirs(download_dir, exist_ok=True)
  
  # 批量下载Blob
  for blob in matching_blobs:
      blob_client = container_client.get_blob_client(blob)
      download_path = os.path.join(download_dir, blob.name)
      with open(download_path, "wb") as download_file:
          download_file.write(blob_client.download_blob().readall())
  

• 进一步优化方向

  • 分页获取Blob：如果匹配的Blob数量极大，可通过max_results参数分页获取，避免内存占用过高：container_client.list_blobs(name_starts_with=target_prefix, max_results=100)
  • 并行下载：使用concurrent.futures.ThreadPoolExecutor实现多线程下载，同时处理多个Blob，大幅提升下载速度：

    from concurrent.futures import ThreadPoolExecutor
    
    def download_blob(blob):
        blob_client = container_client.get_blob_client(blob)
        download_path = os.path.join(download_dir, blob.name)
        with open(download_path, "wb") as download_file:
            download_file.write(blob_client.download_blob().readall())
    
    with ThreadPoolExecutor(max_workers=10) as executor:
        executor.map(download_blob, matching_blobs)
    

内容的提问来源于stack exchange，提问作者Aldrigilivet