Azure Synapse管道运行Spark Notebook时出现文件路径错误

• Spark Notebook手动运行正常，但通过Azure Synapse管道运行时出现文件路径错误
• 代码中使用路径 abfss://storage_name/folder_name/* 加载DataFrame，但管道运行时实际采用路径 abfss://storage_name/filename.parquet，触发路径不存在的报错

{"errorCode":"6002","message":"org.apache.spark.sql.AnalysisException: Path does not exist: abfss://storage_name/filename.parquet\n  at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$checkAndGlobPathIfNecessary$4(DataSource.scala:806)\n\n  at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$checkAndGlobPathIfNecessary$4$adapted(DataSource.scala:803)\n\n  at scala.util.ThreadUtils$.$anonfun$parmap$2(ThreadUtils.scala:372)\n\n  at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)\n\n  at scala.util.Success.$anonfun$map$1(Try.scala:255)\n\n  at scala.util.Success.map(Try.scala:213)\n\n  at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)\n\n  at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)\n\n  at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)\n\n  at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)\n\n  at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)\n\n  at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)\n\n  at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)\n\n  at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)\n\n  at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)","failureType":"UserError","target":"notebook_name","details":[]}

• 检查管道参数配置：若管道给Notebook传递了路径参数，确认参数值未被错误覆盖为单个文件名，确保传递的是folder_name/*格式的路径
• 显式定义完整路径：在Notebook代码中硬编码或从可靠参数获取完整路径，避免依赖管道默认路径上下文，例如：

  df = spark.read.parquet("abfss://storage_name/folder_name/*")
  

• 验证存储权限：确认Synapse管道使用的服务主体/托管标识对folder_name文件夹有读取权限，手动运行可能使用个人权限，管道身份权限不足可能伪装成路径不存在错误
• 验证文件路径：在Notebook中执行以下代码确认目标文件夹下的文件存在：

  dbutils.fs.ls("abfss://storage_name/folder_name/")
  

• 关闭管道路径自动传递：若Notebook活动中启用了"传递文件夹路径"类选项，关闭该选项，保持代码中定义的路径不变

内容的提问来源于stack exchange，提问作者darkstar