如何使用Data Factory v1复制Azure Data Lake Store中的空文件夹？

Great question—this is a common pain point for backup workflows in ADF v1, and yes, you can replicate empty folders with a couple of workarounds since the default behavior skips them when using recursive copy to a file-based sink. Let’s break down the most reliable approaches:

1. Use a Custom PowerShell Activity to Build Empty Folder Structures #

This method directly creates matching empty folder trees in your target Data Lake Store (ADLS) alongside your existing copy activity:

• Step 1: Add a Custom Activity to your ADF v1 pipeline. You’ll need to host your PowerShell script in a storage account and configure the activity to use an Azure Batch account or self-hosted integration runtime (required for executing custom scripts in ADF v1).
• Step 2: Write a PowerShell script that authenticates to Azure, scans your source ADLS for all folders (including empty ones), and recreates them in the target. Here’s a simplified snippet to get you started:

  # Authenticate with your service principal
  $tenantId = "<your-tenant-id>"
  $clientId = "<your-service-principal-client-id>"
  $clientSecret = "<your-service-principal-secret>"
  $subscriptionId = "<your-subscription-id>"
  $sourceAdls = "<source-adls-account-name>"
  $targetAdls = "<target-adls-account-name>"
  $sourceRoot = "/your/source/root/folder"
  $targetRoot = "/your/target/backup/folder"
  
  $secureSecret = ConvertTo-SecureString $clientSecret -AsPlainText -Force
  $cred = New-Object System.Management.Automation.PSCredential($clientId, $secureSecret)
  Connect-AzAccount -ServicePrincipal -Credential $cred -Tenant $tenantId -Subscription $subscriptionId
  
  # Get all folders from the source ADLS
  $allFolders = Get-AzDataLakeStoreChildItem -Account $sourceAdls -Path $sourceRoot -Recurse | Where-Object { $_.Type -eq "Directory" }
  
  # Create each folder in the target if it doesn't exist
  foreach ($folder in $allFolders) {
      $targetPath = $folder.FullName.Replace($sourceRoot, $targetRoot)
      if (-not (Test-AzDataLakeStoreItem -Account $targetAdls -Path $targetPath)) {
          New-AzDataLakeStoreItem -Account $targetAdls -Path $targetPath -Folder
          Write-Host "Created empty folder: $targetPath"
      }
  }
  

• Step 3: Ensure your service principal has Read access to the source ADLS and Write access to the target ADLS so the script can scan and create folders.

2. Temporary Empty File Workaround (Quick Fix for Smaller Structures) #

If your folder hierarchy isn’t too complex, a simpler (though less clean) workaround works:

• Step 1: Run a pre-copy script (via custom activity or Azure Function) to create a tiny temporary file (like .empty) in every empty folder of your source ADLS. This gives the copy activity something to detect, so it’ll replicate the folder.
• Step 2: Execute your existing copy activity with recursive=true—all folders (including previously empty ones) will now be copied to the target.
• Step 3: Run a post-copy script to delete all .empty files from both the source and target ADLS to keep your storage clean.

Pro Tips #

• Run the folder creation script after your copy activity if you want to capture any new folders created mid-workflow. If your source is static during the backup, you can run them in parallel to save time.
• For ADF v1, make sure your custom activity’s runtime is properly configured—Azure Batch is the easiest option for cloud-based execution.

内容的提问来源于stack exchange，提问作者RalJans