如何使用Python SDK处理Azure Storage Blob上传的超时问题？

问题描述 #

使用Python SDK上传Blob到Azure Storage时，无法通过设置timeout参数控制超时行为，不同超时值表现完全一致。SDK版本为azure-storage-blob==12.13.1。

代码示例 #

from azure.storage.blob import BlobServiceClient, ContentSettings
from datetime import datetime
from tenacity import retry, stop_after_attempt, wait_fixed  # 补充缺失的retry依赖导入

STORAGE_CONNECTION_STRING = ''
container = ''
blobFileName = ''
contentType = ''


def upload_blob(localFilePath):
    blob_service_client = BlobServiceClient.from_connection_string(STORAGE_CONNECTION_STRING)
    blob_client = blob_service_client.get_blob_client(container=container, blob=blobFileName)
    content_setting = ContentSettings(content_type=contentType)
    with open(localFilePath, "rb") as data:
        blob_client.upload_blob(data, overwrite=True, content_settings=content_setting, timeout=30)


@retry(Exception, tries=5, delay=1)
def uploadImageToAzure(imagePath):
    try:
        upload_blob(imagePath)
    except Exception as ex:
        print(f'Exception: {ex}, Type:{type(ex)}, Time:{datetime.now()}')
        raise Exception(ex)

    print('Successfully Uploaded the File')
    return True

uploadImageToAzure(imagePath)

测试场景 #

在上传过程中关闭网络，测试不同timeout值的表现：

• timeout=1

Exception: <urllib3.connection.HTTPSConnection object at 0x7fe5d848e9a0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution, Type:<class 'azure.core.exceptions.ServiceRequestError'>, Time:2023-03-28 09:25:42.543262
Exception: <urllib3.connection.HTTPSConnection object at 0x7fe5d83c30a0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution, Type:<class 'azure.core.exceptions.ServiceRequestError'>, Time:2023-03-28 09:27:04.949785
Exception: <urllib3.connection.HTTPSConnection object at 0x7fe5d83da940>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution, Type:<class 'azure.core.exceptions.ServiceRequestError'>, Time:2023-03-28 09:28:28.508035
Exception: <urllib3.connection.HTTPSConnection object at 0x7fe5d837a340>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution, Type:<class 'azure.core.exceptions.ServiceRequestError'>, Time:2023-03-28 09:29:58.473318
Exception: <urllib3.connection.HTTPSConnection object at 0x7fe5d8393be0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution, Type:<class 'azure.core.exceptions.ServiceRequestError'>, Time:2023-03-28 09:31:19.184696

• timeout=2

Exception: <urllib3.connection.HTTPSConnection object at 0x7f19f0164940>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution, Type:<class 'azure.core.exceptions.ServiceRequestError'>, Time:2023-03-28 09:46:25.328502
Exception: <urllib3.connection.HTTPSConnection object at 0x7f19f009b0a0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution, Type:<class 'azure.core.exceptions.ServiceRequestError'>, Time:2023-03-28 09:47:50.919091
Exception: <urllib3.connection.HTTPSConnection object at 0x7f19f00b2940>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution, Type:<class 'azure.core.exceptions.ServiceRequestError'>, Time:2023-03-28 09:49:14.144449
Exception: <urllib3.connection.HTTPSConnection object at 0x7f19f0056340>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution, Type:<class 'azure.core.exceptions.ServiceRequestError'>, Time:2023-03-28 09:50:43.947462
Exception: <urllib3.connection.HTTPSConnection object at 0x7f19f006ebe0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution, Type:<class 'azure.core.exceptions.ServiceRequestError'>, Time:2023-03-28 09:52:05.272197

• timeout=4

Exception: <urllib3.connection.HTTPSConnection object at 0x7f2a08699910>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution, Type:<class 'azure.core.exceptions.ServiceRequestError'>, Time:2023-03-28 09:57:27.401257
Exception: <urllib3.connection.HTTPSConnection object at 0x7f2a085d41f0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution, Type:<class 'azure.core.exceptions.ServiceRequestError'>, Time:2023-03-28 09:58:55.699520
Exception: <urllib3.connection.HTTPSConnection object at 0x7f2a085ea910>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution, Type:<class 'azure.core.exceptions.ServiceRequestError'>, Time:2023-03-28 10:00:18.315710
Exception: <urllib3.connection.HTTPSConnection object at 0x7f2a096ce310>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution, Type:<class 'azure.core.exceptions.ServiceRequestError'>, Time:2023-03-28 10:01:37.800908
Exception: <urllib3.connection.HTTPSConnection object at 0x7f2a096e6bb0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution, Type:<class 'azure.core.exceptions.ServiceRequestError'>, Time:2023-03-28 10:03:02.293558

• timeout=16

Exception: <urllib3.connection.HTTPSConnection object at 0x7ff8c83d7880>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution, Type:<class 'azure.core.exceptions.ServiceRequestError'>, Time:2023-03-28 10:19:15.195291
Exception: <urllib3.connection.HTTPSConnection object at 0x7ff8c830f130>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution, Type:<class 'azure.core.exceptions.ServiceRequestError'>, Time:2023-03-28 10:20:47.769388
Exception: <urllib3.connection.HTTPSConnection object at 0x7ff8c8323880>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution, Type:<class 'azure.core.exceptions.ServiceRequestError'>, Time:2023-03-28 10:22:14.996163
Exception: <urllib3.connection.HTTPSConnection object at 0x7ff8c82c8280>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution, Type:<class 'azure.core.exceptions.ServiceRequestError'>, Time:2023-03-28 10:23:37.736138
Exception: <urllib3.connection.HTTPSConnection object at 0x7ff8c82ddb20>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution, Type:<class 'azure.core.exceptions.ServiceRequestError'>, Time:2023-03-28 10:25:00.846901

原因分析 #

你遇到的ServiceRequestError是DNS解析失败导致的，这个阶段的超时不受Azure SDK的timeout参数控制——SDK的timeout是针对已建立连接后的服务请求超时，而DNS解析属于系统层面的网络操作，遵循操作系统的DNS超时配置。

解决方案 #

1. 配置DNS解析超时 #

修改操作系统的DNS超时设置：

• Linux：调整/etc/resolv.conf中的options timeout:n（n为秒数）
• Windows：通过网卡属性的DNS设置调整超时参数

2. 自定义客户端配置 #

通过azure.core.pipeline设置全局的连接超时（包含DNS解析阶段），示例代码如下：

from azure.storage.blob import BlobServiceClient
from azure.core.pipeline.transport import RequestsTransport
import requests

# 创建自定义transport，设置DNS+TCP连接超时
transport = RequestsTransport(
    session=requests.Session(),
    connection_timeout=1,  # DNS解析+TCP连接总超时，单位秒
    read_timeout=30        # 已连接后的读取超时
)

# 初始化BlobServiceClient时传入自定义配置
blob_service_client = BlobServiceClient.from_connection_string(
    STORAGE_CONNECTION_STRING,
    transport=transport
)

3. 针对性捕获异常处理 #

在代码中捕获DNS解析失败的异常，快速触发重试：

from azure.core.exceptions import ServiceRequestError

def upload_blob(localFilePath):
    try:
        blob_service_client = BlobServiceClient.from_connection_string(STORAGE_CONNECTION_STRING)
        blob_client = blob_service_client.get_blob_client(container=container, blob=blobFileName)
        content_setting = ContentSettings(content_type=contentType)
        with open(localFilePath, "rb") as data:
            blob_client.upload_blob(data, overwrite=True, content_settings=content_setting, timeout=30)
    except ServiceRequestError as e:
        if "Temporary failure in name resolution" in str(e):
            raise  # 直接抛出触发重试

内容的提问来源于stack exchange，提问作者Muhammad Haseeb