AzureML SDKv2任务流Unauthorized错误：DevOps提交时异常

我用AzureML SDKv2构建了AML管道，提交代码如下。任务能在计算集群上正常完成，但调用ml_client.jobs.stream()时出现Unauthorized错误。计算集群已关联托管标识，这个问题只在通过DevOps提交任务时出现，本地手动提交则能正常流式输出日志。

错误栈追踪 #

[2024-05-28 11:30:01Z] Submitting 1 runs, first five are: 156e0e1b:dbc22428-a9da-4cdb-87c2-39ebd8961372
[2024-05-28 11:33:16Z] Completing processing run id dbc22428-a9da-4cdb-87c2-39ebd8961372.
[2024-05-28 11:33:17Z] Submitting 1 runs, first five are: 9f12649a:12a3ce89-f11e-452d-bb9f-0d3c4292778d
WARNING:azure.identity._credentials.chained:ChainedTokenCredential failed to retrieve a token from the included credentials.
Attempted credentials:
    ManagedIdentityCredential: ManagedIdentityCredential authentication unavailable. The requested identity has not been assigned to this resource.
    DefaultAzureCredential: Please run 'az login' to set up an account
ERROR:__main__:Unable to complete the pipeline
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/vsts/work/1/s/ml_service/pipelines/train_pipeline.py", line 308, in <module>
    main(args)
  File "/home/vsts/work/1/s/ml_service/pipelines/train_pipeline.py", line 208, in main
    raise excp
  File "/home/vsts/work/1/s/ml_service/pipelines/train_pipeline.py", line 205, in main
    ml_client.jobs.stream(name=pipeline_run_job.name)
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/azure/ai/ml/_telemetry/activity.py", line 275, in wrapper
    return f(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/azure/ai/ml/operations/_job_operations.py", line 788, in stream
    self._stream_logs_until_completion(
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/azure/ai/ml/operations/_job_ops_helper.py", line 271, in stream_logs_until_completion
    _current_details: RunDetails = run_operations.get_run_details(job_name)

提交代码示例 #

def get_chained_credentials(client_id: str):
    try:
        managed_identity_creds = ManagedIdentityCredential(client_id=client_id)
        default_creds = DefaultAzureCredential()
        chained_creds = ChainedTokenCredential(managed_identity_creds, default_creds)
        return chained_creds
    except Exception as e:
        raise e

ml_client = get_client_workspace(
            get_chained_credentials(e.cluster_identity_id),
            e.subscription_id,
            e.resource_group,
            e.workspace_name,
        )
pipeline_run_job = ml_client.jobs.create_or_update(
                pipeline_job, experiment_name=e.experiment_name_forecast
            )
ml_client.jobs.stream(name=pipeline_run_job.name)

核心原因 #

你当前用计算集群的托管标识创建ML Client，提交作业成功是因为集群自身的标识有执行作业的权限，但jobs.stream()是在DevOps代理进程中运行的——这个进程无法使用计算集群的托管标识（托管标识绑定在集群资源上，仅集群运行时可调用），而DevOps代理环境没有本地az login的上下文，导致DefaultAzureCredential也无法获取有效token，最终认证失败。

方案1：改用DevOps服务主体认证（推荐） #

在DevOps管道中使用服务主体获取凭据，确保代理进程能正常认证：

1. 在Azure中创建服务主体，给它分配Azure Machine Learning工作区参与者权限（或更细粒度的日志读取权限）。
2. 在DevOps管道中添加Azure服务连接，注入以下环境变量：AZURE_TENANT_ID、AZURE_CLIENT_ID、AZURE_CLIENT_SECRET。
3. 修改凭据获取逻辑，优先使用服务主体凭据：

import os
from azure.identity import ClientSecretCredential, ChainedTokenCredential, ManagedIdentityCredential, DefaultAzureCredential

def get_chained_credentials(cluster_client_id: str = None):
    cred_list = []
    # 优先读取DevOps注入的服务主体凭据
    tenant_id = os.getenv("AZURE_TENANT_ID")
    sp_client_id = os.getenv("AZURE_CLIENT_ID")
    sp_secret = os.getenv("AZURE_CLIENT_SECRET")
    if all([tenant_id, sp_client_id, sp_secret]):
        cred_list.append(ClientSecretCredential(
            tenant_id=tenant_id,
            client_id=sp_client_id,
            client_secret=sp_secret
        ))
    # 其次尝试计算集群托管标识（仅集群内运行时生效）
    if cluster_client_id:
        cred_list.append(ManagedIdentityCredential(client_id=cluster_client_id))
    # 最后尝试本地默认凭据
    cred_list.append(DefaultAzureCredential())
    return ChainedTokenCredential(*cred_list)

方案2：调整权限（仅验证用，不推荐） #

如果一定要用计算集群的托管标识，需要确保：

• 给计算集群的托管标识分配工作区的日志读取/作业读取权限
• 注意：DevOps代理进程依然无法直接使用集群的托管标识，这个方案仅适用于你把stream逻辑放到集群运行的作业中，而非DevOps代理进程里。

内容的提问来源于stack exchange，提问作者Obiii