能否从函数返回打开的写入流？S3流式操作报错求助

是否可以从函数返回一个打开的写入流？我运行代码时遇到了这个错误：

ValueError: I/O operation on closed file.

报错代码示例 #

import boto3
import io

s3 = boto3.client('s3')

# 从S3下载文件并上传副本
def download_s3():
    response = s3.get_object(
        Bucket="bedrock-data-files", 
        Key="test/filename.txt"
        )
    stream = io.BytesIO(response['Body'].read())
    return stream

def upload_s3():
    returnedStream = io.BytesIO()
    s3.upload_fileobj(
        returnedStream,
        "bedrock-data-files", 
        "test/copy.txt"
        )
    return returnedStream

downloadstream = download_s3()
uploadstream = upload_s3()
for chunk in downloadstream.read():
    uploadstream.write(chunk)

我有一个可以正常运行的版本，但需要把写入流的创建逻辑放在函数内：

def download_s3():
    response = s3.get_object(
        Bucket="bedrock-data-files", 
        Key="test/filename.txt"
        )
    stream = io.BytesIO(response['Body'].read())
    return stream

downloadstream = download_s3()
print(downloadstream) # 输出 <_io.BytesIO object at 0x10679dbc0>
s3.upload_fileobj(
    downloadstream,
    "bedrock-data-files", 
    "test/copy.txt"
    )

问题原因 #

报错的核心是：s3.upload_fileobj方法在完成上传后会自动关闭传入的流对象。你的upload_s3函数刚创建空流就调用了上传操作，此时流内无数据，上传完成后流被关闭，函数返回的是已关闭的流，后续写入自然触发I/O错误。

解决方案 #

要实现“写入流创建逻辑在函数内”的需求，只需让函数只负责创建并返回写入流，把上传操作放到外部——等写完数据后再执行上传：

修改后的代码示例 #

import boto3
import io

s3 = boto3.client('s3')

def download_s3():
    response = s3.get_object(
        Bucket="bedrock-data-files", 
        Key="test/filename.txt"
        )
    stream = io.BytesIO(response['Body'].read())
    return stream

# 仅创建并返回写入流，不执行上传
def create_upload_stream():
    return io.BytesIO()

downloadstream = download_s3()
uploadstream = create_upload_stream()

# 写入数据到流
uploadstream.write(downloadstream.read())
# 关键：写入后将流指针移到开头，否则upload_fileobj会从末尾读取，导致上传空内容
uploadstream.seek(0)

# 执行上传
s3.upload_fileobj(
    uploadstream,
    "bedrock-data-files", 
    "test/copy.txt"
)

如果想把上传逻辑也封装到函数里，可以让函数接收已写好数据的流作为参数：

def upload_s3_from_stream(data_stream):
    data_stream.seek(0)
    s3.upload_fileobj(
        data_stream,
        "bedrock-data-files", 
        "test/copy.txt"
    )

# 使用方式
downloadstream = download_s3()
uploadstream = create_upload_stream()
uploadstream.write(downloadstream.read())
upload_s3_from_stream(uploadstream)

内容的提问来源于stack exchange，提问作者Jon Wilson