Azure Synapse管道运行Notebook写CSV报6002权限拒绝错误

DevOps 小满

2026-9-3

问题场景

运行平台：Azure Synapse Analytics 工作区/管道
开发语言：PySpark（Python）
故障现象：
- 管道调度运行承载数据转换、存储逻辑的Notebook时触发报错
- 注释掉CSV写入相关代码后，管道可正常运行
- 在Notebook交互运行环境中直接执行CSV写入逻辑无异常，仅管道调度运行时该环节失败
核心报错特征：调用csv写入方法时抛出java.nio.file.AccessDeniedException，403 HEAD请求无权限访问目标ADLS Gen2存储路径，失败类型为UserError，关联管道为Product_Data_pipeline3

完整错误堆栈如下：

{
    "errorCode": "6002",
    "message": "Py4JJavaError: An error occurred while calling o666.csv.\n: java.nio.file.AccessDeniedException: Operation failed: \"This request is not authorized to perform this operation using this permission.\", 403, HEAD, https://bcnpricing.dfs.core.windows.net/test/test/data/output/test_df3.csv?upn=false&action=getStatus&timeout=90\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.checkException(AzureBlobFileSystem.java:1185)\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:504)\n\tat org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1696)\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.exists(AzureBlobFileSystem.java:1013)\n\tat org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:119)\n\tat org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:108)\n\tat org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:106)\n\tat org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:131)\n\tat org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:218)\n\tat org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:256)\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)\n\tat org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:253)\n\tat org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:214)\n\tat org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:148)\n\tat org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:147)\n\tat org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:995)\n\tat org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:107)\n\tat org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:181)\n\tat org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:94)\n\tat org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)\n\tat org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)\n\tat org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:995)\n\tat org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:444)\n\tat org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:416)\n\tat org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:294)\n\tat org.apache.spark.sql.DataFrameWriter.csv(DataFrameWriter.scala:985)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\n\tat py4j.Gateway.invoke(Gateway.java:282)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:238)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: Operation failed: \"This request is not authorized to perform this operation using this permission.\", 403, HEAD, https://bcnpricing.dfs.core.windows.net/test/test/data/output/test_df3.csv?upn=false&action=getStatus&timeout=90\n\tat org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:207)\n\tat org.apache.hadoop.fs.azurebfs.services.AbfsClient.getPathStatus(AbfsClient.java:570)\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getFileStatus(AzureBlobFileSystemStore.java:802)\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:502)\n\t... 35 more\n\nTraceback (most recent call last):\n\n  File \"/opt/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py\", line 1372, in csv\n    self._jwrite.csv(path)\n\n  File \"/home/trusted-service-user/cluster-env/env/lib/python3.8/site-packages/py4j/java_gateway.py\", line 1304, in __call__\n    return_value = get_return_value(\n\n  File \"/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py\", line 111, in deco\n    return f(*a, **kw)\n\n  File \"/home/trusted-service-user/cluster-env/env/lib/python3.8/site-packages/py4j/protocol.py\", line 326, in get_return_value\n    raise Py4JJavaError(\n\npy4j.protocol.Py4JJavaError: An error occurred while calling o666.csv.\n: java.nio.file.AccessDeniedException: Operation failed: \"This request is not authorized to perform this operation using this permission.\", 403, HEAD, https://bcnpricing.dfs.core.windows.net/test/test/data/output/test_df3.csv?upn=false&action=getStatus&timeout=90\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.checkException(AzureBlobFileSystem.java:1185)\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:504)\n\tat org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1696)\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.exists(AzureBlobFileSystem.java:1013)\n\tat org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:119)\n\tat org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:108)\n\tat org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:106)\n\tat org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:131)\n\tat org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:218)\n\tat org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:256)\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)\n\tat org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:253)\n\tat org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:214)\n\tat org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:148)\n\tat org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:147)\n\tat org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:995)\n\tat org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:107)\n\tat org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:181)\n\tat org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:94)\n\tat org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)\n\tat org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)\n\tat org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:995)\n\tat org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:444)\n\tat org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:416)\n\tat org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:294)\n\tat org.apache.spark.sql.DataFrameWriter.csv(DataFrameWriter.scala:985)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\n\tat py4j.Gateway.invoke(Gateway.java:282)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:238)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: Operation failed: \"This request is not authorized to perform this operation using this permission.\", 403, HEAD, https://bcnpricing.dfs.core.windows.net/test/test/data/output/test_df3.csv?upn=false&action=getStatus&timeout=90\n\tat org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:207)\n\tat org.apache.hadoop.fs.azurebfs.services.AbfsClient.getPathStatus(AbfsClient.java:570)\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getFileStatus(AzureBlobFileSystemStore.java:802)\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:502)\n\t... 35 more\n\n",
    "failureType": "UserError",
    "target": "Product_Data_pipeline3",
    "details": []
}

故障根因

两种运行场景采用的访问身份完全不同，权限不匹配是核心原因：

Notebook交互式运行时，默认使用当前登录操作的Azure AD账号身份访问ADLS存储，该账号本身具备目标路径的读写权限，因此执行无异常
管道调度运行Notebook时，默认使用Synapse工作区绑定的系统托管标识(MSI)发起存储访问，报错请求URL中携带的upn=false参数可直接验证：本次请求未传递用户身份信息，完全以托管标识身份访问。该托管标识未被授予目标ADLS路径的对应访问权限，因此触发403 AccessDenied报错。

Spark写入文件前会先发送HEAD请求检查目标路径是否存在，这一步权限校验失败会直接中断流程，不会进入实际写入阶段。

修复方案

按以下步骤配置权限即可解决：

进入Azure Portal，打开报错对应的ADLS Gen2存储账户（域名为bcnpricing.dfs.core.windows.net）
进入存储账户的「访问控制(IAM)」页，点击「添加角色分配」
选择合适的存储角色，将角色分配给Synapse工作区对应的系统托管标识：
- 仅需读写文件：选择「存储Blob数据贡献者」角色即可，满足最小权限要求
- 需要修改路径ACL、管理文件权限：选择「存储Blob数据所有者」角色
由于ADLS Gen2默认开启分层命名空间，还需要进入报错对应的test容器，逐级导航到test/data/output/路径，在路径的「访问ACL」配置中，给上述托管标识追加读取、写入、执行权限，并配置权限向子文件/子目录继承
权限配置完成后等待1-2分钟全局生效，重新触发管道运行即可。

快速校验方法：编辑管道中的Notebook活动，将运行身份设置为「用户身份」（即当前登录管道的用户身份）后试运行，如果运行成功可100%确认是托管标识权限问题。

内容的提问来源于stack exchange，提问作者Jha Ayush