AzureML CLIv2挂载Datastore异常：任务成功但数据集未写入

背景 #

为理解AzureML CLIv2中AzureML与AzureStorage的交互，编写脚本目标是下载MNIST数据集并存储至已配置好的torchvision_data Datastore中。

数据集下载脚本 #

import os
import argparse
import logging
from torchvision.datasets import MNIST,CIFAR10

def main():
    """Main function of the script."""

    # input and output arguments
    parser = argparse.ArgumentParser()
    parser.add_argument("--dst_dir", type=str, help="Directory where to write data")
    parser.add_argument('--dataset_name',type=str,choices=['MNIST','CIFAR10'])
    args = parser.parse_args()

    print(vars(args))
    root_path = os.path.join(args.dst_dir,args.dataset_name)
    if args.dataset_name=="MNIST":
        print(f"Download {args.dataset_name} => {root_path}")
        data_train=MNIST(root=root_path,train=True,download=True)
        data_test=MNIST(root=root_path,train=False,download=True)
    elif args.dataset_name=="CIFAR10":
        print(f"Download {args.dataset_name} => {root_path}")
        data_train=CIFAR10(root=root_path,train=True,download=True)
        data_test=CIFAR10(root=root_path,train=False,download=True)
    else:
        print("Unknown Dataset......")

if __name__ == "__main__":
    main()

任务提交脚本 #

已配置好可用的Environment和Compute Cluster，任务提交入口脚本如下：

from azure.ai.ml import MLClient
from azure.ai.ml import command
from azure.ai.ml import Input, Output
from azure.ai.ml.entities import Environment
from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential
from azure.ai.ml.constants import AssetTypes,InputOutputModes
from datetime import datetime

## => 获取Environment的代码
## => 获取Compute的代码

component_name = f"DataWrapper-{datetime.now().strftime('%Y%m%d%H%M%S')}"
print(component_name)
data_wrapper_component = command(
    name=component_name,
    display_name=component_name,
    description="Download a TorchVision Dataset in AzureStorage...",
    inputs={
        "dataset_name":"MNIST",
    },
    outputs={
        "dst_dir":Output(
            type=AssetTypes.URI_FOLDER,
            folder="azureml://datastores/torchvision_data",
            mode=InputOutputModes.RW_MOUNT),
    },
    # 组件的源文件夹
    code="./code", # 添加整个code文件夹...
    command="""python components/datawrapper/datawrapper.py \
            --dst_dir ${{outputs.dst_dir}}                  \
            --dataset_name ${{inputs.dataset_name}}
            """,
    compute=cpu_compute_target,
    experiment_name="datawrapper",
    is_deterministic=False,
    environment=f"{pipeline_job_env.name}:{pipeline_job_env.version}"
)


returned_job = ml_client.create_or_update(data_wrapper_component)
aml_url = returned_job.studio_url
print("Monitor your job at", aml_url)

问题现象 #

• 任务执行成功，但目标Datastoretorchvision_data中无任何数据。
• 将输出配置中的folder参数替换为path后，任务执行报错。

提问 #

我在挂载输出目录时是否存在操作错误？

内容的提问来源于stack exchange，提问作者FrsECM