Azure Data Factory复制活动Blob Storage接收器超时问题求助

尝试通过Azure Data Factory管道的Copy data activity将Azure Table Storage中的表备份至Azure Blob Storage，多数场景运行正常，但部分存储账户内的大记录量表持续失败，错误提示为：The client could not finish the operation within specified timeout.

排查发现问题源于读取Azure Table Storage时的间歇性延迟：备份过程中大部分请求耗时不足250ms，但偶尔有请求耗时约19000ms，此时Blob上传触发超时，且这类延迟频繁出现导致无法完成备份。

已确认Copy data activity的「General」超时配置为12小时，并非此问题原因；其他存储账户中相同表的备份耗时不到1小时。已尝试单独备份单个表、调整「Maximum Data Integration Units」和「Degree of Copy Parallelism」降低复制速度，均无效果。

错误日志示例：

{    
    "dataRead": 2073796386,    
    "dataWritten": 211593105,    
    "filesWritten": 12,    
    "sourcePeakConnections": 1,    
    "sinkPeakConnections": 1,    
    "rowsRead": 1272000,    
    "rowsCopied": 1272000,    
    "copyDuration": 372,    
    "throughput": 7302.1,    
    "logFilePath": "backup-logs/copyactivity-logs/Copy to Blob Storage/163560cd-ec4b-4fba-86f2-c6a5ea8ce584/",    
    "errors": [        
        {            
            "Code": 9011,            
            "Message": "ErrorCode=UserErrorFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file operation is failed, upload file failed at path: 'backups/20230908/activities/activities_00012.parquet'.,Source=Microsoft.DataTransfer.Common,''Type=Microsoft.WindowsAzure.Storage.StorageException,Message=The client could not finish the operation within specified timeout.,Source=Microsoft.WindowsAzure.Storage,''Type=System.TimeoutException,Message=The client could not finish the operation within specified timeout.,Source=,'",            
            "EventType": 0,            
            "Category": 5,            
            "Data": {},            
            "MsgId": null,            
            "ExceptionType": null,            
            "Source": null,            
            "StackTrace": null,            
            "InnerEventInfos": []        
        }    
    ],    
    "effectiveIntegrationRuntime": "PrivateNetworkIntegrationRuntime (North Central US)",    
    "usedDataIntegrationUnits": 4,    
    "billingReference": {        
        "activityType": "DataMovement",        
        "billableDuration": [            
            {                
                "meterType": "ManagedVNetIR",                
                "duration": 0.4666666666666667,                
                "unit": "DIUHours"            
            }        
        ]    
    },    
    "usedParallelCopies": 1,    
    "executionDetails": [        
        {            
            "source": {                
                "type": "AzureTableStorage"            
            },            
            "sink": {                
                "type": "AzureBlobStorage"            
            },            
            "status": "Failed",            
            "start": "9/8/2023, 10:21:57 AM",            
            "duration": 372,            
            "usedDataIntegrationUnits": 4,            
            "usedParallelCopies": 1,            
            "profile": {                
                "queue": {                    
                    "status": "Completed",                    
                    "duration": 87                
                },                
                "transfer": {                    
                    "status": "Completed",                    
                    "duration": 284,                    
                    "details": {                        
                        "readingFromSource": {                            
                            "type": "AzureTableStorage",                            
                            "workingDuration": 274,                            
                            "timeToFirstByte": 0                        
                        },                        
                        "writingToSink": {                            
                            "type": "AzureBlobStorage",                            
                            "workingDuration": 4                        
                    }                
                }            
            },            
            "detailedDurations": {                
                "queuingDuration": 87,                
                "timeToFirstByte": 0,                
                "transferDuration": 284            
            }        
        }    
    ],    
    "dataConsistencyVerification": {        
        "VerificationResult": "NotVerified"    
    },    
    "durationInQueue": {        
        "integrationRuntimeQueue": 0    
    }
}

1. 配置Blob存储接收器的超时参数 #

在Copy data activity的Azure Blob Storage接收器配置中，通过Advanced选项卡下的Additional properties添加以下键值对，延长上传相关超时：

• writeTimeout: 设置为00:30:00（30分钟），覆盖默认单文件上传超时
• connectionTimeout: 设置为00:05:00（5分钟），调整连接超时阈值

2. 优化Azure Table Storage读取策略 #

• 启用分区发现: 在源配置中开启「Enable partition discovery」，让ADF自动识别表分区键，并行读取不同分区，分散请求压力，降低间歇性延迟的影响
• 调整批量读取大小: 在源的附加属性中设置pageSize（可微调至1000-2000），控制单次读取行数，避免单次请求数据量过大导致延迟
• 添加读取重试: 在源的附加属性中配置maxRetryCount=5和retryInterval=00:00:05，针对延迟请求自动重试，避免单次异常中断任务

3. 调整Copy Activity高级设置 #

• 确保分块上传开启: 确认Blob接收器的「Enable chunking」处于开启状态，大文件拆分为小块上传，单个块失败可自动重试，不会导致整个文件上传失败
• 合理调整DIU与并行度: 尝试适度提高DIU（如至8），同时将并行度设置为2-3，让ADF分配更多资源处理间歇性延迟，避免资源不足引发超时
• 配置活动重试: 在Copy activity的「General」选项卡下，设置「Retry count=3」和「Retry interval=00:01:00」，让整个活动在失败后自动重试

4. 检查网络与Integration Runtime配置 #

• 确认Private Network Integration Runtime与存储账户处于同一区域，避免跨区域网络延迟
• 若使用自托管IR，检查节点的CPU、内存资源是否充足，避免资源瓶颈导致的处理延迟

内容的提问来源于stack exchange，提问作者humbleice