ADF v2中基于管道运行月份动态创建Azure存储子容器的问询

Hey there! Great question—this is a super common scenario in ADF, and we can absolutely make this work. Let's walk through each part of your question and adjust your existing setup step by step.

1. Can we get the pipeline's running month and use it in the pipeline? #

Absolutely! ADF has built-in date functions that let you extract the month from either the tumbling window's start/end time (ideal for your time-based data sync) or the current pipeline run time.

For your use case, using the tumbling window's windowStart parameter makes the most sense—it aligns the subcontainer with the time range of the data you're copying. Here are the key expressions you can use:

• Get the numeric month (e.g., 5 for May): @month(pipeline().parameters.windowStart)
• Get a 2-digit month (e.g., 05 for May): @formatDateTime(pipeline().parameters.windowStart, 'MM')
• Get year + month (e.g., 2024/05 for better organization): @formatDateTime(pipeline().parameters.windowStart, 'yyyy/MM')

2. Where to define the variable (or how to use the value directly)? #

You have two flexible options here:

Option 1: Use the expression directly in the dataset parameter (no separate variable needed) #

This is the simplest approach if you only need the month value for the copy activity. You'll pass the expression directly when referencing the destination dataset.

Option 2: Define a pipeline-level variable (for reuse across multiple activities) #

If you need to use the month value in other parts of the pipeline (e.g., logging, additional copy tasks):

• Go to your pipeline's Variables tab, add a string variable (e.g., targetMonth).
• Add a Set Variable activity at the start of your pipeline, set its value to your chosen month expression (e.g., @month(pipeline().parameters.windowStart)).
• Reference it later with @variables('targetMonth').

3. Is the syntax "dfac/$monthvariable" valid? #

Nope—ADF uses @-prefixed expressions instead of $ for variable/function references. Your path needs to be built using ADF's expression language, either directly in the dataset or via parameters.

Let's adjust your existing Copy activity setup #

Here's how to modify your current configuration to get the dynamic subcontainer:

Step 1: Update your Destination Blob Dataset #

First, add a parameter to your DestinationDataset_ayy to accept the subpath:

1. Open DestinationDataset_ayy (Blob Storage dataset).
2. Go to the Parameters tab, add a string parameter named subContainerPath.
3. In the Connection tab, set the Blob path to:

   @concat('dfac/', dataset().subContainerPath)
   

   This builds the full path like dfac/5 or dfac/2024/05 based on the parameter value.

Step 2: Modify your Copy Activity's output reference #

Update the outputs section of your Copy activity to pass the month value as the parameter. Here's the adjusted JSON snippet:

"outputs": [
    {
        "referenceName": "DestinationDataset_ayy",
        "type": "DatasetReference",
        "parameters": {
            "subContainerPath": "@month(pipeline().parameters.windowStart)"
        }
    }
]

If you went with the variable approach, replace the value with @variables('targetMonth') instead.

Full adjusted Copy activity code (for reference) #

{
    "name": "Copy_ayy",
    "type": "Copy",
    "policy": {
        "timeout": "7.00:00:00",
        "retry": 2,
        "retryIntervalInSeconds": 30,
        "secureOutput": false,
        "secureInput": false
    },
    "userProperties": [
        {
            "name": "Source",
            "value": "/*"
        },
        {
            "name": "Destination",
            "value": "@concat('dfac/', month(pipeline().parameters.windowStart))"
        }
    ],
    "typeProperties": {
        "source": {
            "type": "FileSystemSource",
            "recursive": true
        },
        "sink": {
            "type": "BlobSink",
            "copyBehavior": "PreserveHierarchy"
        },
        "enableStaging": false
    },
    "inputs": [
        {
            "referenceName": "SourceDataset_ayy",
            "type": "DatasetReference",
            "parameters": {
                "cw_modifiedDatetimeStart": "@pipeline().parameters.windowStart",
                "cw_modifiedDatetimeEnd": "@pipeline().parameters.windowEnd"
            }
        }
    ],
    "outputs": [
        {
            "referenceName": "DestinationDataset_ayy",
            "type": "DatasetReference",
            "parameters": {
                "subContainerPath": "@month(pipeline().parameters.windowStart)"
            }
        }
    ]
}

Quick notes #

• If you want leading zeros for months (e.g., 05 instead of 5), use @formatDateTime(pipeline().parameters.windowStart, 'MM') instead of @month(...).
• The PreserveHierarchy copy behavior will keep your source folder structure intact under the dynamic subcontainer, which is perfect for your use case.

内容的提问来源于stack exchange，提问作者aquib.qureshi