Azure Data Factory通过JSON创建数据管道失败问题排查

我希望借助现有Azure Data Factory（ADF）管道的JSON配置，自动化创建仅修改名称等少量内容的新管道。执行Set-AzDataFactoryV2Pipeline命令时，收到“Pipeline Entity not valid”错误；即使直接上传未修改的正常管道JSON也失败。

问题相关代码与信息 #

PowerShell代码

# Retrieve the existing pipeline
$pipeline = Get-AzDataFactoryV2Pipeline -DataFactoryName $dataFactoryName -ResourceGroupName $resourceGroupName -Name $existingPipelineName

# Convert Pipeline configuration to JSON
$pipelineJson = $pipeline | ConvertTo-Json -Depth 10

# Parse the JSON configuration
$pipelineConfig = $pipelineJson | ConvertFrom-Json
# Update the configuration parameters
$pipelineConfig.name = $newPipelineName    

# Convert the updated configuration back to JSON
$updatedPipelineJson = $pipelineConfig | ConvertTo-Json -Depth 20

# Save the updated JSON to a file
$updatedPipelineJson | Out-File -FilePath $jsonFilePath -Encoding utf8

错误信息

Error Code: BadRequest
Error Message: Pipeline Entity not valid
Request Id: 115c7391-7882-41a8-953e-ec1a0b64dcb9
Timestamp (Utc):06/21/2024 14:21:15

生成的JSON示例

{
  "Name": "00_Master_Pipeline",
  "Activities": [
    {
      "DataFlow": {
        "AdditionalProperties": null,
        "ReferenceName": "dataflow1",
        "DatasetParameters": {
          "source1": {},
          "sink1": {}
        },
        "Parameters": {}
      },
      "Staging": {
        "LinkedService": null,
        "FolderPath": null
      },
      "IntegrationRuntime": null,
      "Compute": {
        "ComputeType": "General",
        "CoreCount": 8
      },
      "TraceLevel": "Fine",
      "ContinueOnError": null,
      "RunConcurrently": null,
      "SourceStagingConcurrency": null,
      "LinkedServiceName": null,
      "Policy": {
        "AdditionalProperties": null,
        "Timeout": "0.12:00:00",
        "Retry": 0,
        "RetryIntervalInSeconds": 30,
        "SecureInput": false,
        "SecureOutput": false
      },
      "AdditionalProperties": {
        "typeProperties": {
          "dataflow": {
            "referenceName": "dataflow1",
            "type": "DataFlowReference",
            "parameters": {},
            "datasetParameters": {
              "source1": {},
              "sink1": {}
            }
          },
          "staging": {},
          "compute": {
            "coreCount": 8,
            "computeType": "General"
          },
          "traceLevel": "Fine"
        }
      },
      "Name": "Data",
      "Description": "YYYY",
      "State": null,
      "OnInactiveMarkAs": null,
      "DependsOn": [],
      "UserProperties": []
    }
  ],
  "Parameters": null,
  "Id": "/subscriptions/YYYYY/resourceGroups/YYYYY/providers/Microsoft.DataFactory/factories/YYYYYY/pipelines/00_Master_Pipeline",
  "ETag": "ZZZ",
  "ResourceGroupName": "YYY",
  "DataFactoryName": "XXX"
}

错误原因分析 #

1. 包含只读元数据字段：Get-AzDataFactoryV2Pipeline返回的对象包含ADF服务的只读元数据（如Id、ETag、ResourceGroupName、DataFactoryName），这些字段不属于创建新管道的请求体范畴，ADF会拒绝包含此类字段的请求。
2. 冗余/无效字段：来回转换JSON的过程中，引入了ADF不接受的冗余内容，比如：
   • 活动对象中的AdditionalProperties与顶层字段重复（如DataFlow和AdditionalProperties.typeProperties.dataflow）
   • 大量null值字段（如State、OnInactiveMarkAs、IntegrationRuntime）
3. 结构不符合创建规范：ADF创建管道仅需要核心配置字段（Name、Activities、Parameters、Description等），多余字段会导致服务校验失败。

解决方法 #

直接提取现有管道的核心配置字段，跳过不必要的JSON来回转换，避免引入无效字段：

修正后的PowerShell代码

# 获取现有管道对象
$pipeline = Get-AzDataFactoryV2Pipeline -DataFactoryName $dataFactoryName -ResourceGroupName $resourceGroupName -Name $existingPipelineName

# 仅保留创建新管道所需的核心配置，排除只读元数据
$newPipelineConfig = [PSCustomObject]@{
    Name = $newPipelineName
    Activities = $pipeline.Activities
    Parameters = $pipeline.Parameters
    # 如需保留描述，取消注释下面一行
    # Description = $pipeline.Description
}

# 转换为符合ADF要求的JSON，确保深度足够覆盖嵌套结构
$updatedPipelineJson = $newPipelineConfig | ConvertTo-Json -Depth 20

# 保存JSON到文件
$updatedPipelineJson | Out-File -FilePath $jsonFilePath -Encoding utf8

# 执行创建新管道命令
Set-AzDataFactoryV2Pipeline -DataFactoryName $dataFactoryName -ResourceGroupName $resourceGroupName -Name $newPipelineName -DefinitionFile $jsonFilePath

额外检查点 #

• 生成的JSON中不能包含Id、ETag、ResourceGroupName等元数据字段
• 清理活动配置中的冗余AdditionalProperties字段（如果存在）
• 确保所有null值字段都已被过滤（可通过Select-Object配合-ExcludeProperty进一步优化）

内容的提问来源于stack exchange，提问作者Julian