如何让Python Minio客户端上传断连时触发超时错误？

问题 #

Python Minio客户端向Minio服务器上传文件时，要求连接错误时抛出异常。当前客户端初始连接失败会正常报错，但上传中途断网后进程会无限挂起，希望实现60秒超时后返回错误。

已尝试配置http_client的timeout与重试参数但未生效，而mc命令行工具可正常识别上传中途的断连情况。

简化后的测试代码：

from minio import Minio

client = None
bucket = 'testbucket'

def connect():
    print(f"Creating MinioClient")
    global client
    client = Minio(
        '10.0.0.1:9000',
        access_key='myaccesskey',
        secret_key='mysecretkey',
        secure=False
    )

def createBucket():
    global client
    global bucket
    print(f"Making bucket '{bucket}' if not exists")
    found = client.bucket_exists(bucket)
    if not found:
        client.make_bucket(bucket)
    else:
        print(f"Bucket '{bucket}' already exists")


def putFile():
    """ Upload data """
    print("Uploading the file with fput")

    global client
    global bucket

    objectname = 'ReallyBigFile'
    filepath = '/home/jake/4GBfile.bin'

    try:
        result = client.fput_object(
            bucket,
            objectname,
            filepath,
        )
    except IOError as err:
        print(err)

    print("Created {0} object; etag: {1}, version-id: {2}".format(result.object_name, result.etag, result.version_id))


connect()
createBucket()
putFile()

已尝试的无效配置：

client = Minio(
    '10.0.0.1:9000',
    access_key='myaccesskey',
    secret_key='mysecretkey',
    secure=False,
    http_client=urllib3.poolmanager.PoolManager(
        timeout=1,
        retries=urllib3.Retry(
            total=3,
            backoff_factor=0.2,
            status_forcelist=[500, 502, 503, 504]
        )
    )

解决方案 #

核心修改点 #

• 用urllib3.Timeout明确设置连接超时和读取超时，替代直接传整数的方式
• 扩大异常捕获范围，覆盖Minio客户端及HTTP层的连接类异常
• 针对大文件分片上传，设置合理分片大小配合超时规则

修改后的完整代码 #

from minio import Minio
from minio.error import S3Error
import urllib3
from urllib3.exceptions import ConnectionError as Urllib3ConnectionError

client = None
bucket = 'testbucket'

def connect():
    print(f"Creating MinioClient")
    global client
    # 配置带明确超时规则的HTTP客户端
    http_client = urllib3.PoolManager(
        timeout=urllib3.Timeout(connect=10, read=60),  # 连接超时10秒，读取超时60秒
        retries=urllib3.Retry(
            total=3,
            backoff_factor=0.2,
            status_forcelist=[500, 502, 503, 504],
            allowed_methods=["PUT"]  # 仅允许重试上传类PUT请求
        )
    )
    client = Minio(
        '10.0.0.1:9000',
        access_key='myaccesskey',
        secret_key='mysecretkey',
        secure=False,
        http_client=http_client
    )

def createBucket():
    global client
    global bucket
    print(f"Making bucket '{bucket}' if not exists")
    try:
        found = client.bucket_exists(bucket)
        if not found:
            client.make_bucket(bucket)
        else:
            print(f"Bucket '{bucket}' already exists")
    except (S3Error, Urllib3ConnectionError) as err:
        print(f"Bucket operation failed: {err}")
        raise

def putFile():
    """ Upload data """
    print("Uploading the file with fput")

    global client
    global bucket

    objectname = 'ReallyBigFile'
    filepath = '/home/jake/4GBfile.bin'
    result = None

    try:
        # 设置分片大小，确保每个分片请求遵守超时规则
        result = client.fput_object(
            bucket,
            objectname,
            filepath,
            part_size=10*1024*1024  # 10MB分片，可根据网络情况调整
        )
        print("Created {0} object; etag: {1}, version-id: {2}".format(result.object_name, result.etag, result.version_id))
    except (S3Error, Urllib3ConnectionError, IOError) as err:
        print(f"Upload failed: {err}")
        raise

if __name__ == "__main__":
    try:
        connect()
        createBucket()
        putFile()
    except Exception as err:
        print(f"Program exited with error: {err}")

关键说明 #

• urllib3.Timeout(connect=10, read=60)：connect控制TCP连接建立的超时，read控制从服务器读取响应的超时，上传中途断连会触发read超时
• allowed_methods=["PUT"]：避免重试机制干扰非上传类操作，仅对分片上传的PUT请求生效
• 多类型异常捕获：覆盖Minio服务端错误、HTTP层连接错误及本地IO错误，确保断连时能及时抛出异常并终止进程
• part_size设置：大文件分片上传时，每个分片独立发起请求，合理的分片大小可以让超时机制更灵敏，避免单个分片上传时间过长

内容的提问来源于stack exchange，提问作者JakeRoberts