Spark读取Parquet文件夹报错UnsatisfiedLinkError求助

问题场景 #

• 可正常读取单个Parquet文件：

account_info_df_porsche = spark.read.parquet('./Data/Porsche/Bronze/account_info_Prosche_df_all/part-00000-ea70435f-5dec-4a6e-9903-a51a9523c1ff-c000.snappy.parquet') 

• 读取文件夹内所有Parquet文件时触发Py4JJavaError，核心报错为：

java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z

完整报错栈：

Py4JJavaError: An error occurred while calling o41.parquet.:java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
    at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:793)
    at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:1249)
    at org.apache.hadoop.fs.FileUtil.list(FileUtil.java:1454)
    at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:601)
    at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1972)
    at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:2014)
    at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:761)
    at org.apache.spark.util.HadoopFSUtils$.listLeafFiles(HadoopFSUtils.scala:180)
    at org.apache.spark.util.HadoopFSUtils$.$anonfun$parallelListLeafFilesInternal$1(HadoopFSUtils.scala:95)
    at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at scala.collection.TraversableLike.map(TraversableLike.scala:286)
    at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
    at scala.collection.AbstractTraversable.map(Traversable.scala:108)
    at org.apache.spark.util.HadoopFSUtils$.parallelListLeafFilesInternal(HadoopFSUtils.scala:85)
    at org.apache.spark.util.HadoopFSUtils$.parallelListLeafFiles(HadoopFSUtils.scala:69)
    at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$.bulkListLeafFiles(InMemoryFileIndex.scala:162)
    at org.apache.spark.sql.execution.datasources.InMemoryFileIndex.listLeafFiles(InMemoryFileIndex.scala:133)
    at org.apache.spark.sql.execution.datasources.InMemoryFileIndex.refresh0(InMemoryFileIndex.scala:96)
    at org.apache.spark.sql.execution.datasources.InMemoryFileIndex.<init>(InMemoryFileIndex.scala:68)
    at org.apache.spark.sql.execution.datasources.DataSource.createInMemoryFileIndex(DataSource.scala:539)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:405)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:229)
    at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:211)
    at scala.Option.getOrElse(Option.scala:189)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
    at org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:563)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
    at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
    at java.lang.Thread.run(Unknown Source)

环境信息：Windows系统，Python 3.11.6，Java 17.0.9 LTS，已配置与队友相同的环境变量，队友可正常运行。

解决方法 #

1. 配置Hadoop Windows本地库 #

该错误是Hadoop依赖的Windows本地库缺失或版本不兼容导致的，读取文件夹时需要遍历文件，触发了本地系统调用。

• 下载与Spark依赖的Hadoop版本匹配的Windows本地库（包含winutils.exe和hadoop.dll）。
• 将文件放入自定义Hadoop目录的bin文件夹（例如C:\hadoop\bin）。
• 添加系统环境变量：
  • 新建HADOOP_HOME变量，值为C:\hadoop
  • 将%HADOOP_HOME%\bin添加到PATH变量中
• 重启终端和Python运行环境，使配置生效。

2. 修改Spark会话配置，绕过本地库 #

在创建SparkSession时添加配置，强制使用纯Java实现的文件系统，避免调用本地库：

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("ParquetFolderReader") \
    .config("spark.hadoop.io.native.lib.available", "false") \
    .config("spark.hadoop.fs.file.impl", "org.apache.hadoop.fs.LocalFileSystem") \
    .getOrCreate()

3. 检查文件夹权限 #

确保当前用户对目标文件夹拥有读取和列出文件夹内容的权限：

• 右键目标文件夹 → 属性 → 安全 → 确认当前用户权限列表包含对应权限，无则添加。

4. 验证路径一致性 #

检查代码中文件夹路径的拼写是否与实际文件夹名称一致（注意示例中单个文件路径的account_info_Prosche_df_all与文件夹路径的account_info_Porsche_df_all存在拼写差异，需确认实际路径正确性）。

内容的提问来源于stack exchange，提问作者Miko M