Azure Data Factory自定义活动运行超时问题求助

I've run into similar timeout issues with ADF V1 custom activities targeting blob storage, let's break down your configuration and code to find the root cause and fix it.

Problem Context #

You're following the ADF V1 custom activity documentation, but hitting a timeout error when writing to your output Azure Blob dataset. Below are your current configurations and code:

Pipeline Configuration #

{"name": "ADFTutorialPipelineCustom","properties": {"description": "Use custom activity","activities": [{"type": "DotNetActivity","typeProperties": {"assemblyName": "MyDotNetActivity.dll","entryPoint": "MyDotNetActivityNS.MyDotNetActivity","packageLinkedService": "Destination-BlobStorage","packageFile": "blobcontainer/MyDotNetActivity.zip","extendedProperties": {"SliceStart": "$$Text.Format('{0:yyyyMMddHH-mm}', Time.AddMinutes(SliceStart, 0))"}},"outputs": [{"name": "OutputDataset"}],"policy": {"timeout": "00:30:00","concurrency": 1,"retry": 3},"scheduler": {"frequency": "Minute","interval": 30},"name": "MyDotNetActivity","linkedServiceName": "AzureBatchAccount"}],"start": "2018-01-09T06:00:00Z","end": "2018-01-11T07:00:00Z","isPaused": false,"hubName": "test2017dec11_hub","pipelineMode": "Scheduled"}}

Output Dataset Configuration #

{"name": "OutputDataset","properties": {"published": false,"type": "AzureBlob","linkedServiceName": "Destination-BlobStorage","typeProperties": {"fileName": "{slice}.txt","folderPath": "blobcontainer","partitionedBy": [{"name": "slice","value": {"type": "DateTime","date": "SliceStart","format": "yyyy-MM-dd-HH"}}]},"availability": {"frequency": "Minute","interval": 30}}}

Custom Activity Code #

public class MyDotNetActivity : IDotNetActivity
{
    public IDictionary<string, string> Execute(IEnumerable<LinkedService> linkedServices, IEnumerable<Dataset> datasets, Activity activity, IActivityLogger logger)
    {
        logger.Write("Activity start");
        AzureStorageLinkedService outputLinkedService;
        Dataset outputDataset = datasets.Single(dataset => dataset.Name == activity.Outputs.Single().Name);
        outputLinkedService = linkedServices.First(linkedService => linkedService.Name == outputDataset.Properties.LinkedServiceName).Properties.TypeProperties as AzureStorageLinkedService;
        string connectionString = outputLinkedService.ConnectionString;
        string folderPath = GetFolderPath(outputDataset);
        UploadFileToBlob(connectionString, folderPath, GetFileName(outputDataset), logger);
        return new Dictionary<string, string>();
    }

    private static string GetFolderPath(Dataset dataArtifact)
    {
        if (dataArtifact == null || dataArtifact.Properties == null)
        {
            return null;
        }
        AzureBlobDataset blobDataset = dataArtifact.Properties.TypeProperties as AzureBlobDataset;
        if (blobDataset == null)
        {
            return null;
        }
        return blobDataset.FolderPath;
    }

    public void UploadFileToBlob(string blobConnectionString, string blobFolderPath, string fileName, IActivityLogger logger)
    {
        logger.Write("connecting to the blob..");
        var outputStorageAccount = CloudStorageAccount.Parse(blobConnectionString);
        string output = string.Empty;
        output += "test blob storage";
        var outputBlobUri = new Uri(outputStorageAccount.BlobEndpoint, blobFolderPath + "/" + fileName);
        var outputBlob = new CloudBlockBlob(outputBlobUri, outputStorageAccount.Credentials);
        logger.Write("uploading to the blob URI: {0}", outputBlobUri.ToString());
        outputBlob.UploadText(output);
        logger.Write("upload succeeded");
    }
}

Key Fixes & Troubleshooting Steps #

1. Missing GetFileName Implementation
   Looking at your code, you call GetFileName(outputDataset) but haven't included the implementation for this method. If this method returns null or an invalid filename, the blob upload operation will fail or hang, leading to a timeout. Add the method to correctly retrieve and resolve the filename from your dataset:

   private static string GetFileName(Dataset dataArtifact)
   {
       if (dataArtifact == null || dataArtifact.Properties == null)
       {
           return null;
       }
       AzureBlobDataset blobDataset = dataArtifact.Properties.TypeProperties as AzureBlobDataset;
       if (blobDataset == null)
       {
           return null;
       }
       // Resolve the partitioned {slice} parameter to the actual formatted datetime value
       var sliceValue = ((DateTime)dataArtifact.Properties.PartitionedBy.First(p => p.Name == "slice").Value.Value);
       return blobDataset.FileName.Replace("{slice}", sliceValue.ToString("yyyy-MM-dd-HH"));
   }
   

   Without this, your code is trying to upload to a blob with an invalid filename containing {slice} instead of the actual datetime value.

2. Validate Blob Storage Connection String
   Double-check that the ConnectionString in your Destination-BlobStorage linked service is fully valid (e.g., DefaultEndpointsProtocol=https;AccountName=youraccount;AccountKey=yourkey;EndpointSuffix=core.windows.net). Typos in the account name or key will cause authentication delays or failures that trigger timeouts.

3. Check Network Access to Blob Storage

   • If your Blob Storage account has firewall rules enabled, confirm that your Azure Batch pool's nodes are allowed to access it. You can add the Batch pool's public IP range to the storage firewall exceptions, or deploy both resources in the same virtual network with private endpoints.
   • Verify that Batch nodes have outbound internet access (if using public storage endpoints) — restricted outbound rules in your VNet may block access to the blob service.

4. Inspect Activity Logs for Clues
   Go to the ADF V1 portal's monitoring section, find the failed activity run, and check detailed logs:

   • If you don't see "connecting to the blob..", the issue is in retrieving linked services/datasets (e.g., incorrect names, permission issues with Batch accessing storage to download the activity package).
   • If you see "uploading to the blob URI" but not "upload succeeded", the problem lies with the blob upload itself (network latency, authentication issues, invalid path).

5. Adjust Batch Pool Resources
   If your Batch pool uses small VM sizes (e.g., Standard_A1), the node may lack resources to execute the activity efficiently. Try scaling up the VM size for your Batch pool to see if that resolves the timeout.

6. Test Blob Upload Logic Locally
   Extract the UploadFileToBlob method and test it locally with your storage connection string. This will help you isolate whether the issue is with code logic or the ADF/Batch environment.

内容的提问来源于stack exchange，提问作者Lau Chris