ADF处理后删除ADL文件及邮件通知优化技术问询

Let's tackle both of your ADF pain points with practical, actionable steps—these are common issues folks run into, so I’ve got you covered:

Problem 1: Delete ADL Source Files After ADF Processing (Fixing Access Token Headaches) #

First off, you don’t need to mess with manual REST API calls and curl commands for this—ADF has built-in tools that simplify this, but if you still want to use the API route, I’ll cover both options:

Option 1: Use ADF’s Built-in Delete Activity (Recommended) #

This is the simplest approach, no token management required:

• Add a Delete activity right after your copy activity in the pipeline.
• Configure it to use the same ADLS Gen2 dataset as your source.
• For the file path, use dynamic content to target the exact source file you just processed:

  @activity('YourCopyActivityName').output.sourceFilePath
  

ADF handles all authentication under the hood, so you can skip the entire token acquisition step. Done.

Option 2: Web Activity for REST API (If You Need Custom Logic) #

If you must use the REST API, here’s how to get the access token directly in ADF:

1. Add a "Get Access Token" Web Activity:
   • Method: POST
   • URL: https://login.microsoftonline.com/{your-tenant-id}/oauth2/token
   • Headers: Content-Type: application/x-www-form-urlencoded
   • Body (replace placeholders with your service principal details):

     grant_type=client_credentials&client_id={sp-client-id}&client_secret={sp-secret}&resource=https://storage.azure.com/
     

2. Store the Token in a Variable:
   • Create a string variable (e.g., ADLSToken) and set its value to:

     @activity('GetAccessToken').output.access_token
     

3. Add the Delete API Web Activity:
   • Method: DELETE
   • URL: https://{your-adls-account}.dfs.core.windows.net/{your-file-path}?resource=file
   • Headers:
     • Authorization: Bearer @{variables('ADLSToken')}
     • Content-Type: application/json
       This will authenticate and delete the file without any manual curl steps.

Problem 2: Unified Success/Failure Emails (No More Per-Activity Web Activities) #

Having to configure a web activity for every single failure is messy—here are two clean ways to fix this:

Approach 1: Centralized Status Tracking in the Pipeline #

• Create an Array Variable: Start by adding an array variable (e.g., ActivityStatusLog) to your pipeline.
• Track Each Activity’s Outcome: For each of your 4 main activities, add On Success and On Failure branches:
  • In the Success branch, use a Set Variable activity to append the result to your array:

    @concat(variables('ActivityStatusLog'), '{"Activity":"', activity('MainActivity1').name, '","Status":"Success","Details":"Completed without errors"}')
    

  • In the Failure branch, append the error details:

    @concat(variables('ActivityStatusLog'), '{"Activity":"', activity('MainActivity1').name, '","Status":"Failed","Error":"', activity('MainActivity1').error.message, '"}')
    

• Single Email Web Activity: Add a final Web Activity (calling your Logic App) that runs regardless of prior activity outcomes (use a Set Variable or directly pass the ActivityStatusLog variable in the request body). Your Logic App can then parse this JSON to generate a dynamic, consolidated email.

Approach 2: Logic App Triggered by ADF Pipeline Events (Even Cleaner) #

Skip pipeline-level tracking entirely by letting Logic Apps handle the monitoring:

• In your Logic App, create a trigger: When a pipeline run succeeds or fails (under Azure Data Factory triggers).
• Configure it to target your ADF instance and pipeline.
• The trigger will automatically pull all activity run details (success/failure status, error messages, etc.). You can use Logic App’s built-in JSON parsing and email actions to generate a unified report without touching ADF’s pipeline logic.

内容的提问来源于stack exchange，提问作者Asad Khan