如何查看Azure Data Factory自定义活动的日志输出

我在Azure Data Factory（ADF）管道中用Dotnet开发了一个自定义活动，想通过控制台输出来调试管道。部分预定义活动（比如复制活动）支持将日志导出到存储账户，但自定义活动没有这个复选框选项。我当前用Console.WriteLine输出日志的代码如下：

private static void DeleteBlobFiles(string blobConnectionString, string adlsConnectionString, string blobContainerName, string blobFolderPath, string adlsContainerName, string adlsFolderPath, string adfCustName, string adfSourceSystem, string adfFeedName, string environment)
{
    try
    {
        Console.WriteLine("DeleteBlobFiles method called.");

        string sourceToDestination = "B2R";
        string transactionStatusSuccess = "S";
        string transactionStatusFailure = "F";
        string transactionDescriptionSuccess = "File has been copied to Raw Data Layer of ADLS";
        string transactionDescriptionFailure = "File has not been copied to Raw Data Layer of ADLS";

        CloudStorageAccount storageAccount = CloudStorageAccount.Parse(blobConnectionString);
        CloudBlobClient cloudBlobClient = storageAccount.CreateCloudBlobClient();

        Console.WriteLine("blobContainerName = " + blobContainerName);
        Console.WriteLine("blobFolderPath = " + blobFolderPath);
        // Ensure the container is exist.
        if (!String.IsNullOrEmpty(blobContainerName))
        {
            var blobContainer = cloudBlobClient.GetContainerReference(blobContainerName);
            blobContainer.CreateIfNotExists();
            CloudBlobDirectory cbd = blobContainer.GetDirectoryReference(blobFolderPath);
        }
     }
......
}

我希望为这个自定义活动实现类似预定义复制活动的日志导出功能（将日志持久化到存储账户）。

实现方案 #

方案1：在自定义活动代码中直接写入日志到存储账户

通过Azure Storage SDK，在代码中把日志内容主动写入Blob存储，无需依赖ADF内置选项：

• 给自定义活动新增日志存储相关参数：存储账户连接字符串、日志容器名、日志文件路径
• 封装日志写入方法，同时输出到控制台和Blob存储

修改后的代码示例：

private static void DeleteBlobFiles(string blobConnectionString, string adlsConnectionString, string blobContainerName, string blobFolderPath, string adlsContainerName, string adlsFolderPath, string adfCustName, string adfSourceSystem, string adfFeedName, string environment, string logStorageConnString, string logContainerName, string logFilePath)
{
    // 初始化日志存储客户端
    CloudStorageAccount logStorageAccount = CloudStorageAccount.Parse(logStorageConnString);
    CloudBlobClient logBlobClient = logStorageAccount.CreateCloudBlobClient();
    var logContainer = logBlobClient.GetContainerReference(logContainerName);
    logContainer.CreateIfNotExists();
    var logBlob = logContainer.GetBlockBlobReference($"{logFilePath}/activity_log_{DateTime.UtcNow:yyyyMMddHHmmss}.txt");

    // 封装日志写入逻辑：同时输出到控制台和Blob
    void WriteLog(string message)
    {
        Console.WriteLine(message);
        logBlob.AppendTextAsync($"{DateTime.UtcNow:yyyy-MM-dd HH:mm:ss} - {message}{Environment.NewLine}").Wait();
    }

    try
    {
        WriteLog("DeleteBlobFiles method called.");

        string sourceToDestination = "B2R";
        string transactionStatusSuccess = "S";
        string transactionStatusFailure = "F";
        string transactionDescriptionSuccess = "File has been copied to Raw Data Layer of ADLS";
        string transactionDescriptionFailure = "File has not been copied to Raw Data Layer of ADLS";

        CloudStorageAccount storageAccount = CloudStorageAccount.Parse(blobConnectionString);
        CloudBlobClient cloudBlobClient = storageAccount.CreateCloudBlobClient();

        WriteLog($"blobContainerName = {blobContainerName}");
        WriteLog($"blobFolderPath = {blobFolderPath}");
        
        if (!String.IsNullOrEmpty(blobContainerName))
        {
            var blobContainer = cloudBlobClient.GetContainerReference(blobContainerName);
            blobContainer.CreateIfNotExists();
            CloudBlobDirectory cbd = blobContainer.GetDirectoryReference(blobFolderPath);
        }
    }
    catch (Exception ex)
    {
        WriteLog($"Error occurred: {ex.Message}");
        throw;
    }
}

方案2：利用ADF内置日志+后续活动导出

自定义活动的Console.WriteLine输出会被ADF捕获到活动运行日志中，可直接查看：

1. 运行管道后，进入活动运行详情页
2. 切换到日志标签页，即可看到所有控制台输出

如果需要持久化这些日志，可以添加后续复制活动，将ADF的运行日志导出到指定存储账户。

内容的提问来源于stack exchange，提问作者finisinfinitatis