Azure Databricks用存储账户密钥读取Blob存储报错求助

问题现象 #

使用Blob容器SAS密钥配置可正常读取存储文件，但通过存储账户密钥执行挂载后，读取时触发错误：

Container in account .blob.core.windows.net not found, and we can't create it using anonymous credentials

正常运行的配置 #

spark.conf.set("fs.azure.sas.containername.storageaccount.blob.core.windows.net","sas-key")
dbutils.fs.ls("wasbs://containername@storageaccount.blob.core.windows.net/")
table_name= "main.deltalake_db.Customers_Log_Table_test"
checkpoint_path = "wasbs://testaudit@azdevstoreforlogs.blob.core.windows.net/_checkpoint"
file_path = "wasbs://containername@storageaccount.blob.core.windows.net/topics/logs/"
schema = "xHeaderFields ARRAY<STRUCT<key: STRING, value: STRING>>"
(spark.readStream
  .format("cloudFiles")
  .option("cloudFiles.format", "json")
  .option("cloudFiles.schemaLocation", checkpoint_path)
  .schema(schema)
  .load(file_path)
  .writeStream
  .option("checkpointLocation", checkpoint_path)
  .trigger(availableNow=True)
  .toTable(table_name))

报错的配置 #

dbutils.fs.mount(
source = "wasbs://containername@storageaccount.blob.core.windows.net",
mount_point = "/mnt/topics/logs",
extra_configs = {"fs.azure.account.key.storageaccount.blob.core.windows.net":dbutils.secrets.get(scope="keyvaultscope",key="storage-account-azuredevauditlogs")})
table_name= "main.deltalake_db.Customers_Log_Table_test"
checkpoint_path = "wasbs://testaudit@azdevstoreforlogs.blob.core.windows.net/_checkpoint"
file_path = "wasbs://containername@storageaccount.blob.core.windows.net/topics/logs/"
schema = "xHeaderFields ARRAY<STRUCT<key: STRING, value: STRING>>"
(spark.readStream
  .format("cloudFiles")
  .option("cloudFiles.format", "json")
  .option("cloudFiles.schemaLocation", checkpoint_path)
  .schema(schema)
  .load(file_path)
  .writeStream
  .option("checkpointLocation", checkpoint_path)
  .trigger(availableNow=True)
  .toTable(table_name))

错误原因分析 #

1. 读取路径未匹配挂载规则：虽然执行了挂载操作，但读取时仍使用原始wasbs://路径，挂载仅对/mnt/...格式的挂载点路径生效，Spark尝试匿名访问原始路径导致认证失败。
2. 存储账户密钥未全局生效：挂载时的extra_configs仅作用于当前挂载点，未全局配置Spark的存储账户密钥，直接访问wasbs://路径时无法获取有效认证信息。
3. 挂载有效性未验证：可能挂载操作因密钥错误、容器/存储账户名称拼写错误等原因未成功，但代码未做校验就执行后续读取操作。

解决方法 #

方法1：使用挂载点路径读取文件 #

修改file_path为挂载点路径，替代原始wasbs://路径，完整代码如下：

dbutils.fs.mount(
source = "wasbs://containername@storageaccount.blob.core.windows.net",
mount_point = "/mnt/topics/logs",
extra_configs = {"fs.azure.account.key.storageaccount.blob.core.windows.net":dbutils.secrets.get(scope="keyvaultscope",key="storage-account-azuredevauditlogs")})
# 先验证挂载是否成功
dbutils.fs.ls("/mnt/topics/logs")
table_name= "main.deltalake_db.Customers_Log_Table_test"
checkpoint_path = "wasbs://testaudit@azdevstoreforlogs.blob.core.windows.net/_checkpoint"
# 使用挂载点路径
file_path = "/mnt/topics/logs/topics/logs/"
schema = "xHeaderFields ARRAY<STRUCT<key: STRING, value: STRING>>"
(spark.readStream
  .format("cloudFiles")
  .option("cloudFiles.format", "json")
  .option("cloudFiles.schemaLocation", checkpoint_path)
  .schema(schema)
  .load(file_path)
  .writeStream
  .option("checkpointLocation", checkpoint_path)
  .trigger(availableNow=True)
  .toTable(table_name))

方法2：全局配置存储账户密钥（不依赖挂载） #

如果不需要挂载，直接通过spark.conf全局配置存储账户密钥，和SAS使用方式一致：

spark.conf.set("fs.azure.account.key.storageaccount.blob.core.windows.net", dbutils.secrets.get(scope="keyvaultscope",key="storage-account-azuredevauditlogs"))
dbutils.fs.ls("wasbs://containername@storageaccount.blob.core.windows.net/")
# 后续读取代码保持不变

方法3：先验证挂载有效性再执行读取 #

在挂载后添加校验步骤，确保挂载成功再推进后续操作：

mount_point = "/mnt/topics/logs"
# 先卸载已存在的同名挂载（如果需要）
if any(mount.mountPoint == mount_point for mount in dbutils.fs.mounts()):
    dbutils.fs.unmount(mount_point)
# 执行挂载
dbutils.fs.mount(
source = "wasbs://containername@storageaccount.blob.core.windows.net",
mount_point = mount_point,
extra_configs = {"fs.azure.account.key.storageaccount.blob.core.windows.net":dbutils.secrets.get(scope="keyvaultscope",key="storage-account-azuredevauditlogs")})
# 验证挂载结果
print("挂载验证：", dbutils.fs.ls(mount_point))

内容的提问来源于stack exchange，提问作者ZZZSharePoint