使用boto3 copy结合SourceClient与访问密钥时遇AccessDenied问题

问题场景 #

需在两个需独立密钥访问的S3桶间传输文件，尝试用boto3的copy方法搭配SourceClient参数实现，但执行时触发UploadPartCopy的AccessDenied错误；改用先get_object再put_object的方式则可正常运行。

出错代码 #

def copy_function(src_bucket, src_key, src_akid, src_sak, dest_bucket, dest_key, dest_akid, dest_sak):
    src_client = boto3.client(
        's3',
        aws_access_key_id=src_akid,
        aws_secret_access_key=src_sak
    )

    dest_client = boto3.client(
        's3',
        aws_access_key_id=dest_akid,
        aws_secret_access_key=dest_sak
    )

    rsp = dest_client.copy(
        CopySource={
            'Bucket': src_bucket,
            'Key': src_key
        },
        Bucket=dest_bucket,
        Key=dest_key,
        SourceClient=src_client
    )

错误信息 #

{
  "errorMessage": "An error occurred (AccessDenied) when calling the UploadPartCopy operation: Access Denied",
  "errorType": "ClientError",
  ...
  "stackTrace": [
    "  File \"/var/task/task/lambda_function.py\", line 16, in handler\n    results = test_main(event, context)\n",
    "  File \"/var/task/task/test.py\", line 39, in test_main\n    rsp = dest_client.copy(\n",
    "  File \"/var/task/boto3/s3/inject.py\", line 450, in copy\n    return future.result()\n",
    "  File \"/var/task/s3transfer/futures.py\", line 103, in result\n    return self._coordinator.result()\n",
    "  File \"/var/task/s3transfer/futures.py\", line 266, in result\n    raise self._exception\n",
    "  File \"/var/task/s3transfer/tasks.py\", line 139, in __call__\n    return self._execute_main(kwargs)\n",
    "  File \"/var/task/s3transfer/tasks.py\", line 162, in _execute_main\n    return_value = self._main(**kwargs)\n",
    "  File \"/var/task/s3transfer/copies.py\", line 370, in _main\n    response = client.upload_part_copy(\n",
    "  File \"/var/task/botocore/client.py\", line 565, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n",
    "  File \"/var/task/botocore/client.py\", line 1021, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n"
  ]
}

可正常运行的对比代码 #

def copy_function_2(src_bucket, src_key, src_akid, src_sak, dest_bucket, dest_key, dest_akid, dest_sak):
    src_client = boto3.client(
        's3',
        aws_access_key_id=src_akid,
        aws_secret_access_key=src_sak
    )

    rsp = src_client.get_object(
        Bucket=src_bucket,
        Key=src_key
    )

    dest_client = boto3.client(
        's3',
        aws_access_key_id=dest_akid,
        aws_secret_access_key=dest_sak
    )

    rsp = dest_client.put_object(
        Bucket=dest_bucket,
        Key=dest_key,
        Body=rsp.get('Body').read()
    )

    print(rsp)

核心原因分析 #

两种方法的权限逻辑和操作流程存在本质差异，导致报错：

1. 操作流程差异

   • get_object+put_object：先将源对象下载到本地，再上传到目标桶，仅需源密钥有s3:GetObject、目标密钥有s3:PutObject权限。
   • copy+SourceClient：对于大于5GB的文件会自动触发分块复制（UploadPartCopy），此时由目标客户端发起复制请求，依赖源客户端生成的预签名URL访问源对象，对权限和区域配置的要求更严格。

2. 可能的触发点

   • 区域不匹配：若源桶与目标桶不在同一AWS区域，创建boto3客户端时未指定对应region_name，会导致请求路由错误，触发AccessDenied。
   • 权限遗漏：目标密钥缺少s3:UploadPartCopy权限，或源桶的桶策略限制了UploadPartCopy请求的来源（比如IP、IAM实体限制）。
   • 预签名URL权限不足：源密钥虽能读取源对象，但生成UploadPartCopy所需的预签名URL时，被IAM策略或桶策略限制。

解决方案 #

1. 补充客户端区域配置 #

创建客户端时明确指定对应桶的区域，避免路由错误：

def copy_function(src_bucket, src_key, src_akid, src_sak, src_region, dest_bucket, dest_key, dest_akid, dest_sak, dest_region):
    src_client = boto3.client(
        's3',
        aws_access_key_id=src_akid,
        aws_secret_access_key=src_sak,
        region_name=src_region
    )

    dest_client = boto3.client(
        's3',
        aws_access_key_id=dest_akid,
        aws_secret_access_key=dest_sak,
        region_name=dest_region
    )

    rsp = dest_client.copy(
        CopySource={'Bucket': src_bucket, 'Key': src_key},
        Bucket=dest_bucket,
        Key=dest_key,
        SourceClient=src_client
    )

2. 调整权限策略 #

• 源密钥IAM策略：确保包含s3:GetObject权限：

  {
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::你的源桶名/*"
  }
  

• 目标密钥IAM策略：添加s3:UploadPartCopy权限：

  {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:UploadPartCopy"],
      "Resource": "arn:aws:s3:::你的目标桶名/*"
  }
  

• 源桶桶策略：移除对UploadPartCopy请求的不必要限制，或确保目标密钥的IAM实体被允许访问源对象。

3. 强制单块复制（仅适用于小文件） #

若传输的文件均小于5GB，可通过配置强制boto3使用单块复制，避免触发UploadPartCopy：

from botocore.config import Config

def copy_function(src_bucket, src_key, src_akid, src_sak, dest_bucket, dest_key, dest_akid, dest_sak):
    src_client = boto3.client(
        's3',
        aws_access_key_id=src_akid,
        aws_secret_access_key=src_sak
    )

    # 设置分块阈值大于5GB，强制单块复制
    dest_config = Config(s3={'multipart_threshold': 10 * 1024 * 1024 * 1024})
    dest_client = boto3.client(
        's3',
        aws_access_key_id=dest_akid,
        aws_secret_access_key=dest_sak,
        config=dest_config
    )

    rsp = dest_client.copy(
        CopySource={'Bucket': src_bucket, 'Key': src_key},
        Bucket=dest_bucket,
        Key=dest_key,
        SourceClient=src_client
    )

验证建议 #

优先测试小文件复制：

• 若小文件正常，说明问题出在大文件的分块复制环节，重点检查区域配置和UploadPartCopy权限；
• 若小文件仍报错，优先排查区域配置和源/目标桶的策略限制。

内容的提问来源于stack exchange，提问作者Cogito Ergo Sum