AWS S3 Glacier深度归档对象批量恢复异常及性能问题求助

我把文件归档到AWS S3 Glacier Deep Archive后，需要恢复所有带指定前缀的对象，先后尝试两种方法都遇到了问题：

一、AWS CLI命令仅恢复少量对象 #

最初用以下CLI命令批量发起恢复，但只有少量对象启动了恢复流程，多数失败：

aws s3api list-objects-v2 \
--bucket ${bucket} \
--prefix "${prefix}" \
--query "Contents[?StorageClass=='DEEP_ARCHIVE'].Key" \
--output text \
| sed 's/\t/\n/g' \
| xargs -I %%% \
aws s3api restore-object \
--restore-request Days=${days},GlacierJobParameters={"Tier"=\""${mode}\""} \
--bucket ${bucket} \
--key "%%%"

二、Python脚本处理7万对象效率极低 #

改用Python脚本处理，但针对约70000个对象，脚本运行4小时仍未完成恢复初始化：

def restore_object(bucket,prefix,days,tier):
    s3 = boto3.resource('s3')
    client = boto3.client('s3')

    my_bucket = s3.Bucket(bucket)

    logfile = open("restoration.log","w")

    for object in my_bucket.objects.filter(Prefix=prefix):
        if object.storage_class == "DEEP_ARCHIVE":
            try:
                resp = client.restore_object(
                Bucket=bucket,
                Key=object.key,
                RestoreRequest={
                    'Days' : days,
                    'GlacierJobParameters' : {'Tier' : tier}
                    }
                )
                
            except Exception as e:
                logfile.write(f'For the object {object.key}, {e} \n')

针对CLI命令失败的修复方案 #

• 修正参数格式问题：原命令中--restore-request的JSON引号转义逻辑有误，导致参数解析失败。改用--cli-input-json传递结构化参数更可靠：

  aws s3api list-objects-v2 \
  --bucket ${bucket} \
  --prefix "${prefix}" \
  --query "Contents[?StorageClass=='DEEP_ARCHIVE'].Key" \
  --output text \
  | sed 's/\t/\n/g' \
  | xargs -I %%% \
  aws s3api restore-object \
  --cli-input-json '{
      "Bucket": "'"${bucket}"'",
      "Key": "%%%",
      "RestoreRequest": {
          "Days": '"${days}"',
          "GlacierJobParameters": {
              "Tier": "'"${mode}"'"
          }
      }
  }'
  

• 增加错误收集与并发优化：在命令末尾添加2>> restore_errors.log收集失败请求；用parallel工具替代xargs，提升并发处理能力并支持自动重试。

针对Python脚本效率优化方案 #

• 启用多线程并发处理：单线程遍历7万对象效率极低，用ThreadPoolExecutor实现并发请求，大幅缩短初始化时间：

  import boto3
  from concurrent.futures import ThreadPoolExecutor
  import logging
  
  def restore_single_object(client, bucket, key, days, tier):
      try:
          client.restore_object(
              Bucket=bucket,
              Key=key,
              RestoreRequest={
                  'Days': days,
                  'GlacierJobParameters': {'Tier': tier}
              }
          )
          logging.info(f"已启动恢复：{key}")
      except Exception as e:
          logging.error(f"恢复失败 {key}：{str(e)}")
  
  def restore_objects(bucket, prefix, days, tier, max_workers=50):
      client = boto3.client('s3')
      logging.basicConfig(filename='restoration.log', level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
      
      # 分页获取对象，避免内存过载
      paginator = client.get_paginator('list_objects_v2')
      page_iterator = paginator.paginate(Bucket=bucket, Prefix=prefix)
      
      objects_to_restore = []
      for page in page_iterator:
          if 'Contents' in page:
              objects_to_restore.extend([obj['Key'] for obj in page['Contents'] if obj['StorageClass'] == 'DEEP_ARCHIVE'])
      
      # 并发执行恢复请求
      with ThreadPoolExecutor(max_workers=max_workers) as executor:
          executor.map(lambda key: restore_single_object(client, bucket, key, days, tier), objects_to_restore)
  
  # 调用示例
  restore_objects('你的存储桶名', '指定前缀', 180, 'Standard', max_workers=50)
  

• 优化对象列表获取：用S3分页器替代s3.Bucket.objects.filter，避免一次性加载大量对象；提前过滤符合条件的对象，减少后续处理量。
• 改进日志处理：用Python标准logging模块替代直接写文件，确保日志线程安全且格式规范。

通用注意事项 #

• 避免重复请求：先通过head-object检查对象的Restore字段，跳过已处于恢复中或完成恢复的对象。
• 控制并发数：S3的restore-object有API速率限制，并发数建议设为50-100之间，避免触发限流。
• 确认恢复天数：Deep Archive对象恢复的最低天数为180天，指定Days小于该值会导致请求失败。

内容的提问来源于stack exchange，提问作者Quentin Chartreux