PySpark中使用不同SPN访问同一存储账户的ADLS Gen2容器

核心思路 #

Spark/Hadoop的ABFS协议支持容器级的配置覆盖，只需在配置项中指定容器名称，就能实现同一存储账户下不同容器绑定不同SPN凭据。

具体配置步骤 #

假设你的存储账户名为your-storage-account，需配置的容器为abcd和wxyz，分别对应不同SPN，配置代码如下：

1. 绑定abcd容器的SPN #

# abcd容器专属SPN配置
spark.conf.set("fs.azure.account.auth.type.abcd.your-storage-account.dfs.core.windows.net", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type.abcd.your-storage-account.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id.abcd.your-storage-account.dfs.core.windows.net", "<abcd容器对应SPN的应用ID>")
spark.conf.set("fs.azure.account.oauth2.client.secret.abcd.your-storage-account.dfs.core.windows.net", "<abcd容器对应SPN的密钥>")
spark.conf.set("fs.azure.account.oauth2.client.endpoint.abcd.your-storage-account.dfs.core.windows.net", "https://login.microsoftonline.com/<abcd容器对应SPN的租户ID>/oauth2/token")

2. 绑定wxyz容器的SPN #

# wxyz容器专属SPN配置
spark.conf.set("fs.azure.account.auth.type.wxyz.your-storage-account.dfs.core.windows.net", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type.wxyz.your-storage-account.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id.wxyz.your-storage-account.dfs.core.windows.net", "<wxyz容器对应SPN的应用ID>")
spark.conf.set("fs.azure.account.oauth2.client.secret.wxyz.your-storage-account.dfs.core.windows.net", "<wxyz容器对应SPN的密钥>")
spark.conf.set("fs.azure.account.oauth2.client.endpoint.wxyz.your-storage-account.dfs.core.windows.net", "https://login.microsoftonline.com/<wxyz容器对应SPN的租户ID>/oauth2/token")

验证跨容器读写 #

配置完成后，直接使用标准ABFS路径访问容器即可，Spark会自动匹配对应容器的SPN凭据：

# 读取abcd容器数据
df_abcd = spark.read.csv("abfss://abcd@your-storage-account.dfs.core.windows.net/data/source.csv")

# 写入wxyz容器数据
df_abcd.write.mode("overwrite").parquet("abfss://wxyz@your-storage-account.dfs.core.windows.net/data/output.parquet")

关键说明 #

• 配置格式规则：将容器名称放在存储账户名前方，形成[容器名].[存储账户名].dfs.core.windows.net的域名结构
• 容器级配置优先级高于存储账户级全局配置，会自动覆盖全局设置
• 需确保每个SPN已被授予对应容器的权限（如Storage Blob Data Contributor角色）

内容的提问来源于stack exchange，提问作者Behroz Sikander