使用Python Requests上传文件时文件名路径部分被忽略的问题

使用Python Requests和boto3通过预签名POST URL上传JSON文件时，requests会忽略key的路径部分，S3仅保留文件名的最后一段。示例代码如下：

import boto3, requests, json, tempfile

s3_client = boto3.client('s3')
bucket_name = "some_bucket"
response = s3_client.generate_presigned_post(bucket_name, "${filename}", ExpiresIn=3600)

tmp = tempfile.NamedTemporaryFile()

content = {"hello" : "world"}
json_data = json.dumps(content).encode('utf-8')
key = "this/path/is/ignored/buthisworks.json"

with open(tmp.name, "w") as outfile:
    json.dump(content, outfile)

with open(tmp.name, 'rb') as f:
    files = {'file': (key, f)}
    response = requests.post(response['url'], data=response['fields'], files=files)

实际S3收到的对象名称为buthisworks.json，而非预期的this/path/is/ignored/buthisworks.json。

原因分析 #

当调用generate_presigned_post时使用${filename}占位符，S3会自动提取上传文件的文件名作为对象key。但requests在构造multipart/form-data请求时，会自动截断文件路径，仅保留最后一段文件名，导致路径丢失。

方案1：生成预签名时直接指定完整key路径 #

如果上传的对象路径是固定的，生成预签名时直接传入完整key，无需使用${filename}占位符：

import boto3, requests, json, tempfile

s3_client = boto3.client('s3')
bucket_name = "some_bucket"
key = "this/path/is/ignored/buthisworks.json"
# 直接指定完整key，替换${filename}
response = s3_client.generate_presigned_post(bucket_name, key, ExpiresIn=3600)

tmp = tempfile.NamedTemporaryFile()

content = {"hello" : "world"}
with open(tmp.name, "w") as outfile:
    json.dump(content, outfile)

with open(tmp.name, 'rb') as f:
    # files中的文件名不影响S3对象key，可任意填写
    files = {'file': ('buthisworks.json', f)}
    response = requests.post(response['url'], data=response['fields'], files=files)

方案2：手动覆盖POST数据中的key字段 #

如果需要动态指定对象路径，保留${filename}占位符的同时，在POST请求的data中手动覆盖key字段，传入完整路径：

import boto3, requests, json, tempfile

s3_client = boto3.client('s3')
bucket_name = "some_bucket"
response = s3_client.generate_presigned_post(bucket_name, "${filename}", ExpiresIn=3600)

tmp = tempfile.NamedTemporaryFile()

content = {"hello" : "world"}
key = "this/path/is/ignored/buthisworks.json"

with open(tmp.name, "w") as outfile:
    json.dump(content, outfile)

with open(tmp.name, 'rb') as f:
    # 复制fields并覆盖key字段为完整路径
    post_data = response['fields'].copy()
    post_data['key'] = key
    # files中的文件名不影响，S3会使用post_data中指定的key
    files = {'file': (key, f)}
    response = requests.post(response['url'], data=post_data, files=files)

内容的提问来源于stack exchange，提问作者Raj