运行AzureML管道时出现无效数据绑定表达式错误

问题场景 #

执行命令提交包含超参数调优的AzureML Pipeline时触发无效数据绑定错误，报错指向inputs.data、outputs.model_output、search_space.batch_size、search_space.learning_rate绑定无效：

run_id=$(az ml job create -f path_to_pipeline/pipeline.yaml --query name -o tsv -g grp_name -w ws-name)

相关配置与代码 #

pipeline.yaml原配置

$schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json
type: pipeline
display_name: pipeline_with_hyperparameter_sweep
description: Tune hyperparameters
settings:
  default_compute: azureml:compute-name  # sub with your compute name
jobs:
  sweep_step:
    type: sweep
    inputs:
      data:
        type: uri_file
        path: azureml:code_train_data:1  #data store I created
    outputs:
      model_output:
    sampling_algorithm: random
    search_space:
      batch_size:
        type: choice
        values: [1, 5, 10, 15]
      learning_rate:
        type: loguniform
        min_value: -6.90775527898 # ln(0.001)
        max_value: -2.30258509299 # ln(0.1)
    trial:
      code: ../src
      command: >-
        python train.py 
        --data_path ${{inputs.data}} 
        --output_path ${{outputs.model_output}} 
        --batch_size ${{search_space.batch_size}} 
        --learning_rate ${{search_space.learning_rate}}
      environment: azureml:env_finetune_component:1
    objective:
      goal: maximize
      primary_metric: bleu_score
    limits:
      max_total_trials: 5
      max_concurrent_trials: 3
      timeout: 3600
      trial_timeout: 720

train.py原代码

import argparse

def main(args):
    pass

def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_arguments("--data_path")
    parser.add_arguments("--output_path")
    parser.add_arguments("--batch_size", type=int)
    parser.add_arguments("--learning_rate", type=float)
    args = parser.parse_args()

    return args


if __name__ == "__main__":
    args = parse_args()
    main(args)

Azure CLI版本

{
  "azure-cli": "2.53.0",
  "azure-cli-core": "2.53.0",
  "azure-cli-telemetry": "1.1.0",
  "extensions": {
    "ml": "2.20.0"
  }
}

错误原因 #

1. Trial作业未独立定义IO：Sweep作为父级作业，其inputs/outputs无法直接被子级trial作业引用，trial需要显式定义自身的输入输出，并与父级Sweep的IO绑定。
2. 参数解析语法错误：train.py中误用parser.add_arguments（复数），正确方法应为parser.add_argument（单数），虽不影响绑定校验，但会导致后续脚本执行失败。
3. Trial类型未显式声明：未指定trial为command类型作业，导致AzureML无法正确解析绑定表达式规则。

解决方法 #

1. 修正pipeline.yaml配置 #

调整trial部分，补充类型声明、独立IO定义及父级绑定：

$schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json
type: pipeline
display_name: pipeline_with_hyperparameter_sweep
description: Tune hyperparameters
settings:
  default_compute: azureml:compute-name  # 替换为你的计算集群名称
jobs:
  sweep_step:
    type: sweep
    inputs:
      data:
        type: uri_file
        path: azureml:code_train_data:1  # 你的数据集
    outputs:
      model_output:
    sampling_algorithm: random
    search_space:
      batch_size:
        type: choice
        values: [1, 5, 10, 15]
      learning_rate:
        type: loguniform
        min_value: -6.90775527898 # ln(0.001)
        max_value: -2.30258509299 # ln(0.1)
    trial:
      type: command  # 显式声明trial为command类型作业
      inputs:
        # 将父级sweep的data输入绑定到trial的data输入
        data: ${{parent.inputs.data}}
      outputs:
        # 将父级sweep的model_output输出绑定到trial的model_output输出
        model_output: ${{parent.outputs.model_output}}
      code: ../src
      command: >-
        python train.py 
        --data_path ${{inputs.data}} 
        --output_path ${{outputs.model_output}} 
        --batch_size ${{search_space.batch_size}} 
        --learning_rate ${{search_space.learning_rate}}
      environment: azureml:env_finetune_component:1
    objective:
      goal: maximize
      primary_metric: bleu_score
    limits:
      max_total_trials: 5
      max_concurrent_trials: 3
      timeout: 3600
      trial_timeout: 720

2. 修正train.py参数解析 #

将add_arguments改为add_argument：

import argparse

def main(args):
    pass

def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument("--data_path")
    parser.add_argument("--output_path")
    parser.add_argument("--batch_size", type=int)
    parser.add_argument("--learning_rate", type=float)
    args = parser.parse_args()

    return args


if __name__ == "__main__":
    args = parse_args()
    main(args)

3. 重新提交作业 #

执行原提交命令即可正常触发超参数调优Pipeline。

内容的提问来源于stack exchange，提问作者matsuo_basho