Azure Data Lake本地服务器至数据湖存储增量复制任务咨询

Hey Vipin, great question! Incrementally syncing local server files to Azure Data Lake Storage (ADLS) is a common use case, and there are a few solid approaches you can take depending on your needs—whether you prefer a low-code enterprise tool, a lightweight command-line solution, or a custom script. Let's break them down:

ADF is Azure's fully managed data integration service, perfect for scheduled, scalable incremental syncs. Here's how to set it up for your 3 folders:

• Step 1: Set up linked services

  • First, create an Azure Data Factory instance in your Azure subscription.
  • Create a Self-Hosted Integration Runtime and install it on your local server—this lets ADF access your local files.
  • Create two linked services: one for your local file system (using the self-hosted IR) and one for your ADLS Gen2 account (use a service principal or account key for authentication).

• Step 2: Define datasets

  • For each of your 3 local folders, create a local file system dataset pointing to the folder path.
  • Create corresponding ADLS Gen2 datasets, each pointing to the target container/path where you want to sync the files.

• Step 3: Build an incremental copy pipeline

  • Use a Copy Activity in your pipeline: set the source to your local dataset, and the sink to your ADLS dataset.
  • To enable incremental sync, go to the Source tab of the Copy Activity, under "Incremental copy" select Last modified date. Set a start date (for the initial full sync, use a very old date like 1900-01-01).
  • Alternatively, use a Get Metadata Activity to fetch all files in the local folder, then a Filter Activity to keep only files where the LastModified timestamp is newer than the last pipeline run time. Pass these filtered files to the Copy Activity.
  • Add a Schedule Trigger (e.g., daily, hourly) to automate the incremental sync runs.

AzCopy is Microsoft's fast, command-line tool for transferring data to/from Azure storage. It's ideal if you want a simple, minimal-code solution:

• First full sync
  Run this command for each of your 3 folders to copy all initial files:

  azcopy copy "C:\YourLocalFolder1\*" "https://youradlsaccount.dfs.core.windows.net/yourcontainer/target-folder1" --recursive
  

• Incremental sync (subsequent runs)
  Use the --sync flag—this compares source and target files, only copying new/modified files and skipping unchanged ones:

  azcopy sync "C:\YourLocalFolder1\*" "https://youradlsaccount.dfs.core.windows.net/yourcontainer/target-folder1" --recursive
  

  You can wrap these commands in a batch script (Windows) or shell script (Linux) and schedule it via Task Scheduler (Windows) or cron (Linux) to run automatically at your desired interval.

If you need granular control over the sync logic (e.g., custom filtering, logging), a PowerShell script is a great option. Here's a basic example you can adapt for your 3 folders:

# Configuration parameters
$localFolder = "C:\YourLocalFolder1"
$adlsAccount = "your-adls-account-name"
$adlsContainer = "your-container"
$adlsTargetPath = "synced-files/folder1/"
$lastSyncLog = "C:\SyncLogs\last-sync-folder1.txt"

# Get last sync time (or use a very old date for first run)
if (Test-Path $lastSyncLog) {
    $lastSyncTime = Get-Content $lastSyncLog | Get-Date
} else {
    $lastSyncTime = [DateTime]::MinValue
}

# Fetch files modified after last sync
$filesToUpload = Get-ChildItem -Path $localFolder -Recurse -File | 
                 Where-Object { $_.LastWriteTime -gt $lastSyncTime }

# Upload each file to ADLS
foreach ($file in $filesToUpload) {
    # Convert local file path to ADLS path format
    $adlsFilePath = $adlsTargetPath + $file.FullName.Substring($localFolder.Length + 1).Replace("\", "/")
    Set-AzDataLakeGen2ItemContent -AccountName $adlsAccount -Container $adlsContainer -Path $adlsFilePath -File $file.FullName
    Write-Host "Uploaded: $($file.Name)"
}

# Update last sync time log
Get-Date | Out-File $lastSyncLog

You can duplicate this script for each of your 3 folders, adjust the parameters, and schedule it via Task Scheduler.

Quick Notes: #

• For ADF's self-hosted IR, ensure the server has network access to Azure, and the IR service account has read permissions on your local folders.
• AzCopy's --sync flag uses file last modified time and size by default—add --check-md5=true if you want to verify file content for changes.
• Whichever method you choose, test the incremental sync with a small subset of files first to ensure it works as expected!

内容的提问来源于stack exchange，提问作者Vipinkumar Jha