Azure Synapse中PySpark读写ADLS时LinkedServiceName配置异常排查

API 探路者

2026-7-2

Azure Synapse PySpark读取ADLS认证失败：LinkedServiceName配置问题

问题描述

在Azure Synapse中使用PySpark读取ADLS文件时，触发认证失败错误，提示No LinkedServiceName provided in configuration. Please set spark.storage.synapse.linkedServiceName。通过spark.conf.get()能获取对应存储账户的LinkedServiceName，但spark.sparkContext.getConf().get()无返回结果，怀疑配置未正确生效。

配置代码

spark.conf.set("fs.azure.account.oauth.provider.type", "com.microsoft.azure.synapse.tokenlibrary.LinkedServiceBasedTokenProvider")
spark.conf.set(f"spark.storage.synapse.{source_full_storage_account_name}.linkedServiceName", linked_service)
spark.sparkContext._jsc.hadoopConfiguration().set(f"spark.storage.synapse.{source_full_storage_account_name}.linkedServiceName", linked_service)
spark.sparkContext._jsc.hadoopConfiguration().set("spark.storage.synapse.linkedServiceName", "LS_abs_idodatalakedev")

adls_path = f"abfs://{container}@{storage_account}.dfs.core.windows.net/"
file = adls_path + file_location
spark.read.format('csv').load(file)

错误信息

Py4JJavaError Traceback (most recent call last) Cell In [53], line 7
5 adls_path = f"abfs://{container}@{storage_account}.dfs.core.windows.net/"
6 file = adls_path + file_location
----> 7 spark.read.format('csv').load(file)
File /opt/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py:177,
in DataFrameReader.load(self, path, format, schema, **options)
175 self.options(**options)
176 if isinstance(path, str):
--> 177 return self._df(self._jreader.load(path))
178 elif path is not None:
179 if type(path) != list:
File
~/cluster-env/clonedenv/lib/python3.10/site-packages/py4j/java_gateway.py:1321, in JavaMember.call(self, *args) 1315 command =
proto.CALL_COMMAND_NAME +\ 1316 self.command_header +\ 1317
args_command +\ 1318 proto.END_COMMAND_PART 1320 answer =
self.gateway_client.send_command(command)
-> 1321 return_value = get_return_value( 1322 answer, self.gateway_client, self.target_id, self.name) 1324 for temp_arg
in temp_args: 1325 temp_arg._detach()
File /opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py:190, in
capture_sql_exception..deco(*a, **kw)
188 def deco(*a: Any, **kw: Any) -> Any:
189 try:
--> 190 return f(*a, **kw)
191 except Py4JJavaError as e:
192 converted = convert_exception(e.java_exception)
File
~/cluster-env/clonedenv/lib/python3.10/site-packages/py4j/protocol.py:326,
in get_return_value(answer, gateway_client, target_id, name)
324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
325 if answer[1] == REFERENCE_TYPE:
--> 326 raise Py4JJavaError(
327 "An error occurred while calling {0}{1}{2}.
".
328 format(target_id, ".", name), value)
329 else:
330 raise Py4JError(
331 "An error occurred while calling {0}{1}{2}. Trace:
{3}
".
332 format(target_id, ".", name, value))
Py4JJavaError: An error occurred while calling o4037.load. : Status
code: -1 error code: null error message: Auth failure: HTTP Error
-1CustomTokenProvider getAccessToken threw org.apache.hadoop.fs.azurebfs.contracts.exceptions.KeyProviderException
: No LinkedServiceName provided in configuration. Please set
spark.storage.synapse.linkedServiceNameorg.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator$HttpException:
HTTP Error -1CustomTokenProvider getAccessToken threw
org.apache.hadoop.fs.azurebfs.contracts.exceptions.KeyProviderException
: No LinkedServiceName provided in configuration. Please set
spark.storage.synapse.linkedServiceName at
org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(AbfsRestOperation.java:274)
...

排查情况

执行spark.conf.get(f"spark.storage.synapse.{source_full_storage_account_name}.linkedServiceName")能正常获取配置的LinkedServiceName
执行spark.sparkContext.getConf().get(f"spark.storage.synapse.{source_full_storage_account_name}.linkedServiceName")无返回结果

问题分析与解决方案

核心原因

spark.conf和spark.sparkContext.getConf()属于不同配置上下文：

spark.conf是Spark SQL专属配置，仅对SQL操作生效；但ADLS的ABFS驱动基于Hadoop生态，只会读取Hadoop Configuration
你虽尝试写入Hadoop配置，但可能存在两个问题：
1. 存储账户名参数不匹配：source_full_storage_account_name和访问路径中的storage_account是否完全一致（需为纯账户名，不能带.dfs.core.windows.net后缀）
2. 配置优先级冲突：同时设置特定账户和全局LinkedServiceName时，ABFS驱动优先查找对应账户的配置，若账户名不匹配则报错

修复步骤

统一存储账户名参数
确认source_full_storage_account_name为纯存储账户名（如mystorageaccount），且与构建adls_path时的storage_account完全一致。
统一配置到Hadoop Configuration
放弃spark.conf.set()，所有相关配置直接写入Hadoop Configuration，因为ABFS驱动仅读取该配置：

# 设置OAuth提供者类型
spark.sparkContext._jsc.hadoopConfiguration().set(
    "fs.azure.account.oauth.provider.type", 
    "com.microsoft.azure.synapse.tokenlibrary.LinkedServiceBasedTokenProvider"
)
# 设置对应存储账户的LinkedServiceName
spark.sparkContext._jsc.hadoopConfiguration().set(
    f"spark.storage.synapse.{storage_account}.linkedServiceName", 
    linked_service
)
# 可选：全局默认配置，仅在无特定账户配置时生效
spark.sparkContext._jsc.hadoopConfiguration().set(
    "spark.storage.synapse.linkedServiceName", 
    "LS_abs_idodatalakedev"
)

adls_path = f"abfs://{container}@{storage_account}.dfs.core.windows.net/"
file = adls_path + file_location
spark.read.format('csv').load(file)

验证配置生效状态
通过以下代码确认Hadoop配置已正确设置：

# 检查特定账户配置
print(spark.sparkContext._jsc.hadoopConfiguration().get(f"spark.storage.synapse.{storage_account}.linkedServiceName"))
# 检查全局配置
print(spark.sparkContext._jsc.hadoopConfiguration().get("spark.storage.synapse.linkedServiceName"))

简化配置（可选）
若仅访问单个存储账户，直接使用全局配置即可，无需设置特定账户的配置：

spark.sparkContext._jsc.hadoopConfiguration().set(
    "fs.azure.account.oauth.provider.type", 
    "com.microsoft.azure.synapse.tokenlibrary.LinkedServiceBasedTokenProvider"
)
spark.sparkContext._jsc.hadoopConfiguration().set(
    "spark.storage.synapse.linkedServiceName", 
    "LS_abs_idodatalakedev"
)

内容的提问来源于stack exchange，提问作者Herwini

Azure Synapse中PySpark读写ADLS时LinkedServiceName配置异常排查

问题描述 #

配置代码 #

错误信息 #

排查情况 #

问题分析与解决方案 #

核心原因 #

修复步骤 #