Azure技术问题：如何从数据湖复制分区数据并读取各文件夹文件

Got it, let's break down how to set up your Copy Activity to read all files across every countryName/year folder under /raw/project/. Here are a couple of practical approaches depending on your needs:

1. Quick & Simple: Use Wildcards in Folder Path #

If you just need to pull all files from every country and year folder without targeting specific values, this is the easiest way:

• In your source dataset, set the folderPath using an expression with wildcards. The * acts as a placeholder for any folder name at that level:

  "folderPath": {
      "value": "/raw/project/*/*/",
      "type": "Expression"
  }
  

• Make sure to set the File name field to *.json so you only pick up your JSON files.
• Also, double-check that the Recursively option in the dataset's Settings is enabled (it’s usually on by default, but confirming ensures it grabs all files inside each year folder).

2. Dynamic & Targeted: Use ForEach with Parameters #

If you need to iterate over a specific list of countries/years (instead of all of them), use nested ForEach activities with pipeline parameters:

• First, create two pipeline parameters:
  • countryList (array type): e.g., ["Germany", "France", "Spain"]
  • yearList (array type): e.g., ["2021", "2022", "2023"]
• Add an outer ForEach activity, set its Items to @pipeline().parameters.countryList, and name it something like LoopCountries.
• Inside LoopCountries, add an inner ForEach activity, set its Items to @pipeline().parameters.yearList, name it LoopYears.
• Inside LoopYears, add your Copy Activity. For the source dataset's folderPath, use this expression to dynamically plug in the current country and year:

  "folderPath": {
      "value": "/raw/project/@{parent().item()}/@{item()}/",
      "type": "Expression"
  }
  

  (Note: @parent().item() refers to the current country from the outer loop, @item() refers to the current year in the inner loop.)

Bonus: Capture Folder Metadata in Your Output #

If you want to include the country and year values as columns in your target data (super useful for partitioning later), use the Additional columns feature in the Copy Activity's Source tab:

• Add a column named country with the expression: @split(item().folderPath, '/')[3] (adjust the index if your path structure varies)
• Add a column named year with the expression: @split(item().folderPath, '/')[4]

内容的提问来源于stack exchange，提问作者gabi