Azure SDKv2注册URI File后读取数据报HTTP Error 409问题

问题背景 #

我在Azure ML中将Blob存储的数据注册为URI File，目的是实现数据版本控制并保留数据沿袭（组件要求输入为URI File或URI Folder）。注册过程成功，但直接通过返回的URL读取数据时，因存储账户禁止公网访问，触发PublicAccessNotPermitted错误，无法读取数据。

已执行的操作与错误信息 #

1. 注册URI File数据资产 #

# Setting the credentials
subscription_id = 'my_id'
resource_group = 'my_group'
workspace_name = 'my_workspace'

workspace = Workspace(subscription_id, resource_group, workspace_name)

# authenticate
credential = DefaultAzureCredential()
credential.get_token("https://management.azure.com/.default")
# Get a handle to the workspace
ml_client = MLClient(
    credential=credential,
    subscription_id=subscription_id,
    resource_group_name=resource_group,
    workspace_name=workspace_name,
)

# Waking up "lazy" client
data_asset = ml_client.data.get(name="coffee_data", version=1)
print(f"Data asset URI: {data_asset.path}")

from azure.ai.ml.entities import Data
from azure.ai.ml.constants import AssetTypes

web_path = 'https://containername12345.blob.core.windows.net/my-container/raw_data/raw.csv'

raw_data = Data(
    name="TESTESTEST",
    path=web_path,
    type=AssetTypes.URI_FILE,
    description="Dataset for some work I do!",
    tags={"user":"Egor"},
)

data = ml_client.data.create_or_update(raw_data)
print(
    f"Dataset with name {data.name} was registered to workspace, the dataset version is {data.version}"
)

输出：

Dataset with name TESTESTEST was registered to workspace, the dataset version is 4

2. 获取数据资产并尝试读取 #

data_asset = ml_client.data.get(name="TESTESTEST", version = ml_client.data._get_latest_version('TESTESTEST').version)
print(f"Data asset URI: {data_asset.path}")

输出：

Data asset URI: https://containername12345.blob.core.windows.net/my-container/raw_data/raw.csv

直接访问该URL时返回错误：

This XML file does not appear to have any style information associated with it. The document tree is shown below.
<Error>
<Code>PublicAccessNotPermitted</Code>
<Message>Public access is not permitted on this storage account. RequestId:abcde-12345 Time:2023-08-18T07:31:07.6896441Z</Message>
</Error>

使用pandas读取时触发报错：

df = pd.read_csv(data_asset.path)

报错信息：

---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
Cell In[8], line 1
----> 1 df = pd.read_csv(data_asset.path)

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/pandas/io/parsers/readers.py:912, in read_csv(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, date_format, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, encoding_errors, dialect, on_bad_lines, delim_whitespace, low_memory, memory_map, float_precision, storage_options, dtype_backend)
    899 kwds_defaults = _refine_defaults_read(
    900     dialect,
    901     delimiter,
   (...)
    908     dtype_backend=dtype_backend,
    909 )
    910 kwds.update(kwds_defaults)
--> 912 return _read(filepath_or_buffer, kwds)

... (省略多行)

HTTPError: HTTP Error 409: Public access is not permitted on this storage account.

解决方案 #

以下三种方法可解决该读取问题，根据场景选择：

方法1：通过Azure ML SDK下载到本地后读取 #

利用ml_client.data.download()方法将数据资产下载到本地路径，再读取本地文件：

# 下载数据资产到指定本地路径
local_file_path = ml_client.data.download(
    name="TESTESTEST",
    version=data.version,
    download_path="./temp_data"
)
# 读取本地文件
df = pd.read_csv(local_file_path)

方法2：生成带SAS Token的访问URL #

通过Azure Blob Storage SDK生成临时SAS Token，拼接成可访问的URL后读取：

from azure.storage.blob import BlobServiceClient
from datetime import datetime, timedelta

# 初始化Blob服务客户端
blob_service_client = BlobServiceClient(
    account_url="https://containername12345.blob.core.windows.net",
    credential=credential
)
container_client = blob_service_client.get_container_client("my-container")
blob_client = container_client.get_blob_client("raw_data/raw.csv")

# 生成有效期1小时的只读SAS Token
sas_token = blob_client.generate_shared_access_signature(
    permission="r",
    expiry=datetime.utcnow() + timedelta(hours=1)
)
# 拼接带SAS的访问URL
sas_url = f"{web_path}?{sas_token}"
# 读取数据
df = pd.read_csv(sas_url)

方法3：利用托管标识直接访问Blob（Azure ML环境内） #

如果在Azure ML计算实例/集群中运行代码，可直接使用工作区托管标识认证，无需额外凭证：

from azure.storage.blob import BlobClient

# 从数据资产URL初始化Blob客户端，使用默认凭证（托管标识）
blob_client = BlobClient.from_blob_url(
    data_asset.path,
    credential=DefaultAzureCredential()
)
# 读取Blob流到DataFrame
with blob_client.open_read() as blob_stream:
    df = pd.read_csv(blob_stream)

场景说明 #

• 方法1：适合需要本地文件副本的场景，操作简单，无需额外SDK依赖
• 方法2：适合需要临时共享访问URL的场景，可灵活控制权限和有效期
• 方法3：适合Azure ML内部环境运行，利用托管标识自动认证，无需管理SAS Token

内容的提问来源于stack exchange，提问作者Egorsky