如何在Azure Data Factory复制活动中保存API响应头至变量或文件？

我使用Azure Data Factory（ADF）的复制活动从API GET请求拉取数据，已将响应体保存至存储账户，但需要保存或查看API返回的HTTP响应头，请问是否有办法将这些响应头保存到变量或文件中？

ADF的复制活动本身不会直接返回或保存HTTP响应头——从你提供的复制活动执行响应可以看到，结果里只有数据传输相关的统计指标，没有响应头字段。要实现获取并保存响应头的需求，推荐两种方案：

方案1：Web活动 + 后续处理（推荐） #

这是最简洁的实现方式，利用ADF原生的Web活动捕获响应头：

• 添加Web活动：调用目标API的GET请求，Web活动的执行结果中会包含ADFWebActivityResponseHeaders字段（如你提供的Web活动示例所示），该字段完整存储了API返回的HTTP响应头。
• 存入变量：在Web活动后添加设置变量活动，使用表达式 @activity('你的Web活动名称').output.ADFWebActivityResponseHeaders 将响应头存入一个对象类型的变量中。
• 保存到文件：通过复制活动或存储过程活动，将变量中的响应头序列化为JSON字符串后写入存储账户（如Blob存储），示例表达式：@string(variables('你的响应头变量名'))。
• 处理响应体：如果需要继续处理API的响应体，可在Web活动后添加复制活动，将@activity('你的Web活动名称').output.value作为数据源，写入目标存储。

方案2：自定义活动（进阶） #

如果需要更复杂的业务逻辑处理，可以使用ADF的自定义活动，编写C#或Python代码调用API，同时捕获响应体和响应头，再分别保存到存储账户。这种方式需要额外的代码开发和集成运行时配置，适合特殊场景。

复制活动执行响应示例 #

{
    "dataRead": 99,
    "dataWritten": 99,
    "filesWritten": 1,
    "sourcePeakConnections": 1,
    "sinkPeakConnections": 1,
    "rowsRead": 1,
    "rowsCopied": 1,
    "copyDuration": 10,
    "throughput": 0.01,
    "errors": [],
    "effectiveIntegrationRuntime": "AutoResolveIntegrationRuntime",
    "usedDataIntegrationUnits": 1,
    "billingReference": {
        "activityType": "DataMovement",
        "billableDuration": [
            {
                "meterType": "AzureIR",
                "duration": 1,
                "unit": "DIUHours"
            }
        ],
        "totalBillableDuration": [
            {
                "meterType": "AzureIR",
                "duration": 1,
                "unit": "DIUHours"
            }
        ]
    },
    "usedParallelCopies": 1,
    "executionDetails": [
        {
            "source": {
                "type": "RestService"
            },
            "sink": {
                "type": "AzureBlobFS",
                "region": "Europe"
            },
            "status": "Succeeded",
            "start": "12/12/2023, 2:22:22 PM",
            "duration": 14,
            "usedDataIntegrationUnits": 1,
            "usedParallelCopies": 1,
            "profile": {
                "queue": {
                    "status": "Completed",
                    "duration": 10
                },
                "transfer": {
                    "status": "Completed",
                    "duration": 10,
                    "details": {
                        "readingFromSource": {
                            "type": "RestService",
                            "workingDuration": 10,
                            "timeToFirstByte": 1
                        },
                        "writingToSink": {
                            "type": "AzureBlobFS",
                            "workingDuration": 0
                        }
                    }
                }
            },
            "detailedDurations": {
                "queuingDuration": 10,
                "timeToFirstByte": 3,
                "transferDuration": 2
            }
        }
    ],
    "dataConsistencyVerification": {
        "VerificationResult": "NotVerified"
    },
    "durationInQueue": {
        "integrationRuntimeQueue": 0
    }
}

Web活动执行响应示例 #

{
  "value": [],
  "ADFWebActivityResponseHeaders": {
      "Vary": "Accept-Encoding",
      "x-ms-ratelimit-remaining-tenant-reads": "11999",
      "Strict-Transport-Security": "max-age=31536000; includeSubDomains",
      "x-ms-correlation-request-id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "session-id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "x-ms-correlation-id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "x-ms-client-request-id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "x-ms-request-id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "x-ms-ratelimit-microsoft.consumption-retry-after": "300",
      "x-ms-ratelimit-remaining-microsoft.consumption-requests": "100",
      "X-Content-Type-Options": "nosniff",
      "x-ms-routing-request-id": "EUROPE:20231212T121212Z:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "Cache-Control": "private",
      "Date": "Tue, 12 Dec 2023 12:12:12 GMT",
      "Content-Length": "12",
      "Content-Type": "application/json; charset=utf-8"
  },
  "effectiveIntegrationRuntime": "AutoResolveIntegrationRuntime",
  "executionDuration": 1,
  "durationInQueue": {
      "integrationRuntimeQueue": 0
  },
  "billingReference": {
      "activityType": "ExternalActivity",
      "billableDuration": [
          {
              "meterType": "AzureIR",
              "duration": 1,
              "unit": "Hours"
          }
      ]
  }
}

内容的提问来源于stack exchange，提问作者sixplus4iszen